GAO 


United  States  Government  Accountability  Office _ 

Report  to  Congressional  Committees 


DEFENSE 

MANAGEMENT 

Guidance  and 
Progress  Measures 
Are  Needed  to  Realize 
Benefits  from  Changes 
in  DOD’s  Joint 
Requirements  Process 


GAO-12-339 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

FEB  2012  2.  REPORT  TYPE 

3.  DATES  COVERED 

00-00-2012  to  00-00-2012 

4.  TITLE  AND  SUBTITLE 

Defense  Management:  Guidance  and  Progress  Measures  Are  Needed  to 
Realize  Benefits  from  Changes  in  DOD’s  Joint  Requirements  Process 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

U.S.  Government  Accountability  Office, 441  G  Street 

NW, Washington, DC, 20548 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

_ _ _  ABSTRACT 

18.  NUMBER  19a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  OS 

unclassified  unclassified  unclassified  Report  (SAR) 

49 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


February  2012 


GAO 

Accountability  *  Integrity  *  Reliability 

Highlights 


Highlights  of  GAO-12-339,  a  report  to 
congressional  committees 


DEFENSE  MANAGEMENT 

Guidance  and  Progress  Measures  Are  Needed  to 
Realize  Benefits  from  Changes  in  DOD’s  Joint 
Requirements  Process 


Why  GAO  Did  This  Study 


What  GAO  Found 


The  Department  of  Defense’s  Joint 
Requirements  Oversight  Council 
(JROC)  is  charged  with  assisting  in  the 
prioritization  of  capability  needs  from  a 
joint  perspective  and  helping  guide 
investments.  The  JROC  is  supported 
by  the  Joint  Capabilities  Integration 
and  Development  System  (JCIDS) 
process.  However,  a  congressional 
committee  and  GAO  have  expressed 
concerns  about  the  extent  to  which 
JCIDS  has  been  effective  in  prioritizing 
capability  needs.  The  Ike  Skelton 
National  Defense  Authorization  Act  for 
Fiscal  Year  201 1  required  GAO  to 
provide  a  report  on  the  effectiveness  of 
JCIDS  in  several  areas.  In  addition  to 
responding  to  this  direction,  GAO  has 
more  broadly  evaluated  the  extent  to 
which  (1 )  the  Joint  Staff  has  developed 
and  implemented  an  analytic  approach 
to  prioritize  capability  needs  and  (2) 
the  JROC  has  considered  aspects  of 
the  availability  and  operational  support 
of  weapon  systems — called 
sustainment — when  validating  the 
requirements  of  proposed  capability 
solutions.  To  do  so,  GAO  analyzed 
capability  documents,  reviewed 
relevant  guidance  and  law,  and 
interviewed  officials. 

What  GAO  Recommends 

GAO  recommends  that  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff 
(1)  revise  and  implement  JCIDS 
guidance  to  reflect  recent  changes  to 
the  process  and  establish  criteria  and 
measures  for  determining  the  relative 
importance  of  capability  needs  and  (2) 
require  program  sponsors  to  address 
each  criterion  in  JCIDS  guidance 
related  to  sustainment  in  capability 
documents.  DOD  partially  concurred 
with  GAO’s  recommendations. 


After  studying  the  Joint  Capabilities  Integration  and  Development  System 
(JCIDS)  process  since  September  2010,  the  Joint  Staff  began  initiating  actions  in 
October  201 1  to  better  prioritize  capability  needs  and  align  those  needs  with 
available  budgetary  resources.  Specifically,  according  to  Joint  Staff  officials,  the 
Joint  Requirements  Oversight  Council  (JROC)  has  begun  to  consider  the 
benefits  and  affordability  of  new  capabilities  within  the  context  of  joint  capability 
areas  and  to  evaluate  possible  duplication  before  validating  new  capability 
requirements.  The  Joint  Staff  has  begun  to  implement  a  new  approach  to  support 
JROC  prioritization  of  capability  needs,  but  the  new  approach  is  still  evolving  and 
has  not  been  fully  developed  and  clearly  documented.  New  guidance  does  not 
clearly  outline  goals  of  the  new  approach,  develop  and  communicate  the  analytic 
approach  envisioned  to  support  JROC  decision  making,  or  set  out  criteria  and 
accompanying  measures  of  progress.  GAO  previously  reported  that  JCIDS’s 
ability  to  prioritize  needs  could  be  improved  if  it  had  an  analytic  approach  to 
reviewing  and  validating  proposals  that  would  help  ensure  that  the  most 
important  capability  needs  of  the  department  are  addressed.  Until  the  Joint  Staff 
takes  steps  to  fully  develop,  document,  and  institutionalize  the  new  analytic 
approach,  it  is  not  clear  whether  the  current  momentum  for  improving  the  JCIDS 
process  will  be  sustained. 


JCIDS  guidance  in  effect  through  December  201 1  required  that  sponsors  of 
potential  major  defense  acquisition  programs  address  sustainment  information  in 
capability  development  documents  according  to  four  metrics — materiel 
availability,  operational  availability,  reliability,  and  ownership  cost.  Each  of  these 
metrics  includes  a  set  of  potentially  reportable  criteria  or  data,  which  are  listed  as 
review  criteria  and  are  suggested,  but  not  clearly  required  by  the  guidance,  to  be 
included  in  the  metric.  Based  on  GAO’s  analysis  of  six  capability  development 
documents,  GAO  found  that  all  of  the  documents  provided  information  on  the 
four  required  sustainment  metrics,  but  the  completeness  of  information  for  all  of 
the  metrics’  key  criteria  varied.  Further,  in  some  cases  information  that  should 
have  been  included,  according  to  Department  of  Defense  officials,  was  not 
provided.  The  Joint  Staff  issued  updated  JCIDS  guidance  in  January  2012,  but 
the  guidance  still  does  not  clearly  require  program  sponsors  to  report  on  the 
individual  criteria  for  each  of  the  four  sustainment  metrics.  Without  complete  and 
detailed  information  on  each  of  the  individual  criteria  elements,  the  JROC  may 
not  have  the  information  it  needs  to  make  the  most  informed  decisions  when 
validating  the  requirements  of  proposed  solutions  intended  to  mitigate  capability 
gaps. 
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GAO 


United  States  Government  Accountability  Office 
Washington,  DC  20548 


February  24,  2012 
Congressional  Committees 

Like  the  rest  of  the  federal  government,  the  Department  of  Defense 
(DOD)  is  operating  in  a  constrained  budget  environment  and  facing 
difficult  decisions  about  how  to  invest  its  resources  to  meet  national 
security  objectives.  The  Joint  Requirements  Oversight  Council  (JROC) 
directly  and  indirectly  supports  the  Chairman  of  the  Joint  Chiefs  of  Staffs 
role  as  the  principal  military  advisor  to  the  President  and  other  senior 
national  security  officials.  The  JROC  consists  of  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff  and  other  senior  military  officials  and,  among  other 
duties,  is  expected  to  assist  in  the  prioritization  of  joint  military 
requirements  and  ensure  that  resource  levels  needed  to  fulfill  those 
requirements  are  consistent  with  the  level  of  priority.  These  duties  can 
help  ensure  that  proposed  weapon  systems  are  prioritized  to  create  an 
affordable  portfolio.  The  JROC  uses  the  Joint  Capabilities  Integration  and 
Development  System  (JCIDS),  a  process  created  in  2003  to  guide  the 
development  of  capabilities  from  a  joint  perspective,  to  help  it  identify 
capability  gaps  and  validate  the  requirements  of  proposed  capability 
solutions  to  mitigate  those  gaps.  JCIDS  operates  in  conjunction  with  two 
other  DOD  processes — the  acquisition  process,  which  facilitates  the 
development  and  acquisition  of  proposed  capabilities,  and  the  budgeting 
process,  through  which  these  capabilities  are  funded.  The  development 
and  acquisition  of  weapon  systems  consume  a  substantial  portion  of 
DOD’s  budget.  From  fiscal  years  201 1  through  2016,  DOD  plans  to  spend 
almost  $390  billion  on  its  current  portfolio  of  major  defense  acquisition 
programs.1  Moreover,  DOD  spends  billions  of  dollars  each  year  to  sustain 
these  weapon  systems.  As  we  have  reported,  costs  to  operate  and 
support  these  systems  make  up  at  least  70  percent  of  a  system’s  life 
cycle  costs.2  The  House  Armed  Services  Committee  and  GAO  have 


1  Major  defense  acquisition  programs  are  those  programs  identified  by  DOD  that  are 
estimated  to  require  eventual  total  expenditure  for  research,  development,  test,  and 
evaluation,  including  all  planned  increments,  of  more  than  $365  million  or  expenditure  for 
procurement,  including  all  planned  increments,  of  more  than  $2.19  billion  in  fiscal  year 
2000  constant  dollars.  Programs  may  also  be  designated  by  DOD  as  major  defense 
acquisition  programs. 

2GAO,  Defense  Management:  DOD  Needs  Better  Information  and  Guidance  to  More 
Effectively  Manage  and  Reduce  Operating  and  Support  Costs  of  Major  Weapon  Systems , 
GAO-10-717  (Washington,  D.C.:  July  20,  2010). 
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expressed  concerns  about  the  extent  to  which  JCIDS  has  been  effective 
in  assigning  priorities  to  capabilities  proposed  to  meet  the  needs  of  the 
joint  force. 

A  provision  in  the  Ike  Skelton  National  Defense  Authorization  Act  for 
Fiscal  Year  201 1  required  GAO  to  provide  a  report  on  the  effectiveness  of 
JCIDS  in  several  areas,  such  as  prioritizing  joint  requirements  and 
considering  information  on  costs  of  sustaining  future  programs.3  In 
appendix  I,  we  have  reprinted  the  elements  of  the  provision  to  be  covered 
by  the  report  and  our  response  to  each  one.4  As  part  of  our  analysis,  we 
also  evaluated  (1)  the  extent  to  which  the  Joint  Staff  has  developed  and 
implemented  an  analytic  approach  to  prioritize  capability  needs  and 
(2)  the  extent  to  which  the  JROC  has  considered  aspects  of  the 
availability  and  operational  support  requirements  of  weapon  systems — 
called  sustainment — when  validating  the  requirements  of  proposed 
capability  solutions. 

To  assess  the  extent  to  which  the  Joint  Staff  has  developed  and 
implemented  an  analytic  approach  to  prioritize  capability  needs,  we 
reviewed  relevant  law  and  Joint  Staff  policy  documents  and  related 
guidance  on  the  roles  and  responsibilities  of  the  JCIDS  process  and  the 
JROC  as  it  pertains  to  prioritization.  We  interviewed  Joint  Staff  officials  to 
discuss  changes  to  its  approach  for  managing  requirements  within 
capability  portfolios  and  making  recommendations  for  trade-offs  among 
alternatives.  We  also  reviewed  prior  GAO  reports  that  discussed 
capability  development  and  requirements  prioritization  and  compared  the 
current  efforts  to  minimize  risks  and  identify  unnecessary  overlap  and 
duplication  with  what  we  reported  in  the  past.  To  assess  the  extent  to 
which  the  JROC  has  considered  aspects  of  the  availability  and 
operational  support  requirements  of  weapon  systems — called 
sustainment — when  validating  the  requirements  of  proposed  capability 
solutions,  we  reviewed  DOD  and  Joint  Staff  policy  documents  and  related 
guidance  on  the  requirement  to  develop  sustainment  metrics  for  capability 
documents  for  certain  programs  processed  through  JCIDS.  We 
interviewed  DOD  and  Joint  Staff  officials  to  discuss  the  preparation, 
presentation,  and  consideration  of  sustainment  data.  We  also  selected  a 


3See  Pub.  L.  No.  111-383,  §  862  (2011). 

4We  provided  a  draft  of  this  report  to  the  congressional  defense  committees  on  January  6, 
2012,  to  satisfy  this  requirement. 
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nonprobability  sample  of  six  requirements  documents  for  programs  that 
have  been  required  to  report  sustainment  metrics  to  determine  whether 
the  documents  contained  the  required  metrics  and  the  supporting 
information  included  with  the  metrics. 

We  conducted  this  performance  audit  from  April  201 1  through  February 
2012  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives.  Additional  details  on  our 
scope  and  methodology  are  in  appendix  II. 


Background 


JCIDS;  the  Planning,  Programming,  Budgeting  and  Execution  process; 
and  the  Defense  Acquisition  System  broadly  make  up  DOD’s  overall 
defense  acquisition  management  framework.  JCIDS  was  implemented  in 
2003  to  guide  future  defense  programs  from  a  joint  capabilities 
perspective.  JCIDS  is  one  of  the  first  steps  in  DOD’s  acquisition 
processes;  JCIDS  participants  work  to  identify  and  determine  whether  to 
validate  the  need  for  capabilities  proposed  by  the  services,  the  defense 
agencies,  and  the  combatant  commands.  Once  a  requirement  is 
validated,  the  services  rely  on  the  DOD’s  Planning,  Programming, 
Budgeting  and  Execution  process,  through  which  DOD  allocates  financial 
resources  across  the  department — including  the  services — to  identify 
funding  for  validated  capability  solutions.  DOD  then  manages  the 
development  and  procurement  of  proposed  capabilities  through  the 
Defense  Acquisition  System. 

DOD  implemented  the  JCIDS  process  in  2003  in  an  effort  to  assist  the 
JROC  by  changing  DOD’s  requirements  validation  process  from  a 
service-specific  perspective  to  a  joint  capabilities  perspective.  The  JROC, 
which  is  chaired  by  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff, 
consists  of  a  general  or  admiral  from  each  of  the  military  services  and 
may  include  combatant  commanders  or  deputy  commanders  when 
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directed  by  the  JROC  Chairman.5  The  JROC  is  charged  with  assisting  the 
Chairman  of  the  Joint  Chiefs  of  Staff  with  a  number  of  tasks,  including 
(1)  identifying,  assessing,  and  approving  joint  military  requirements  to 
meet  the  national  military  strategy;  (2)  establishing  and  assigning  priority 
levels  for  joint  military  requirements;  and  (3)  reviewing  the  estimated  level 
of  resources  required  to  fulfill  each  joint  military  requirement  and  ensuring 
that  the  resource  level  is  consistent  with  the  requirement’s  priority,  among 
others.  The  JROC  also  assists  acquisition  officials  in  identifying 
alternatives  to  any  acquisition  programs  that  meet  joint  military 
requirements  for  the  purposes  of  certain  statutory  provisions,  addressing 
matters  such  as  cost  growth.  In  2009,  the  Weapon  Systems  Acquisition 
Reform  Act  expanded  the  role  of  the  JROC  by  directing  it  to  assist  the 
Chairman  of  the  Joint  Chiefs  of  Staff  in  ensuring  that  trade-offs  among 
cost,  schedule,  and  performance  objectives  are  considered  for  joint 
military  requirements  and  establishing  an  objective  for  the  overall  period 
of  time  within  which  an  initial  operational  capability  should  be  delivered  to 
meet  each  joint  military  requirement.6 

The  JROC  reviews  requirements  for  programs  designated  as  JROC- 
interest  based  on  their  expected  cost  and  complexity7  and,  under 


5The  following  officials  serve  as  advisors  to  the  JROC  on  matters  within  their  authority  and 
expertise:  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics;  the 
Under  Secretary  of  Defense  (Comptroller);  the  Under  Secretary  of  Defense  for  Policy;  the 
Director  of  Cost  Assessment  and  Program  Evaluation;  the  Director  of  Operational  Test 
and  Evaluation;  and  such  other  civilian  officials  of  DOD  as  designated  by  the  Secretary  of 
Defense.  See  10  U.S.C.  §  181(d)(1).  The  JROC  must  also  seek  and  consider  input  from 
the  commanders  of  the  combatant  commands  in  carrying  out  several  of  its  missions.  See 
§  181(d)(2). 

6See  Pub.  L.  No.  111-23,  §§  201(b)(1)(C),  201(b)(4)  (2009)  (codified  at  10  U.S.C.  §  181). 
Both  actions  are  to  be  taken  in  consultation  with  certain  officials.  See  id. 

7The  JROC-interest  designation  applies  to  all  programs  that  are  estimated  to  require 
eventual  total  expenditures  for  research,  development,  test,  and  evaluation,  including  all 
planned  increments,  of  more  than  $365  million  or  expenditures  for  procurement,  including 
all  planned  increments,  of  more  than  $2.19  billion  in  fiscal  year  2000  constant  dollars,  or 
that  are  designated  as  major  defense  acquisition  programs  or  special  interest.  The  JROC- 
interest  designation  also  applies  to  major  automated  information  systems  programs,  which 
are  programs  designated  as  such  or  estimated  to  exceed  certain  expenditure  thresholds. 
Finally,  the  JROC-interest  designation  applies  to  all  joint  doctrine,  organization,  training, 
materiel,  leadership  and  education,  personnel,  facilities,  and  policy  change 
recommendations  and  programs  and  capabilities  that  have  a  potentially  significant  impact 
on  interoperability  in  allied  and  coalition  partner  operations.  For  JROC-interest  documents, 
recommendations  on  capability  needs  are  passed  on  to  the  Joint  Capabilities  Board  that 
reviews  and,  if  appropriate,  forwards  them  with  recommendations  to  the  JROC  for 
validation  and  approval. 
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guidance  in  effect  through  December  201 1 ,  also  reviewed  programs  at 
the  request  of  certain  senior  DOD  officials.  Within  JCIDS,  the  JROC  is 
supported  in  its  duty  to  review  and  validate  joint  capability  needs  by  the 
Joint  Capabilities  Board  and  six  Functional  Capabilities  Boards.8  The 
Joint  Capabilities  Board  is  chaired  by  the  Director  of  the  Joint  Staffs 
Directorate  for  Force  Structure,  Resources,  and  Assessment,  and  each 
Functional  Capabilities  Board  is  chaired  by  a  general/flag  officer  or  civilian 
equivalent.  The  Joint  Capabilities  Board  reviews  capability  documents 
before  they  are  passed  on  to  the  JROC  for  its  review  and  also  serves  as 
the  validation  authority  for  certain  programs  that  do  not  reach  JROC- 
interest  thresholds,9  although  the  JROC  may  review  any  JCIDS  document 
or  other  issues  requiring  joint  resolution.  Functional  Capabilities  Boards 
are  responsible  for  reviewing  proposed  requirements  specific  to  joint 
capability  areas,  such  as  protection,  logistics,  or  battlespace  awareness. 

In  JCIDS,  the  JROC  and  its  supporting  organizations  review  requirements 
documents  related  to  capability  gaps  and  the  major  defense  acquisition 
programs  intended  to  fill  those  gaps  prior  to  key  acquisition  milestones. 
These  requirements  documents — initial  capabilities  documents,  capability 
development  documents,  and  capability  production  documents  for 
materiel  solutions  and  change  recommendations  for  nonmateriel 
solutions — are  submitted  into  the  JCIDS  process  by  capability  sponsors.10 
The  initial  capabilities  document  identifies  a  specific  capability  gap,  or  set 


functional  Capabilities  Boards  assess  needs  and  make  recommendations  about 
validating  capability  gaps  for  documents  to  be  validated  by  the  JROC  or  the  Joint 
Capabilities  Board. 

9Under  the  JCIDS  guidance  prior  to  January  10,  2012,  the  Joint  Capabilities  Board  served 
as  the  validation  authority  for  Joint  Capabilities  Board-interest  programs,  which  were 
defined  as  all  acquisition  category  II  and  below  programs  where  the  capabilities,  systems, 
or  both  associated  with  the  document  would  have  affected  the  joint  force  and  an 
expanded  joint  review  was  required.  See  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 
31 70.01  G  at  GL-6  (Mar.  1 , 2009)  (hereinafter  cited  as  CJCSI  31 70.01  G  (Mar.  1 , 2009)). 
Under  the  current  version  of  the  guidance,  Joint  Capabilities  Board-interest  documents  are 
those  describing  acquisition  category  II  and  below  programs  that  have  a  potentially 
significant  impact  on  interoperability  (interagency,  allied,  partner  nation,  coalition,  etc.). 

See  Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development 
System,  enc.  C,  para.  3(e)(2)  (Jan.  19,  2012)  (hereinafter  cited  as  JCIDS  Manual  (Jan.  19, 
2012)).  Under  both  the  previous  and  present  versions  of  the  guidance,  programs  that  are 
not  designated  as  JROC-  or  Joint  Capabilities  Board-interest  are  validated  by  the  sponsor. 
See  JCIDS  Manual,  enc.  C,  para.  3(e)(3)-(5)  (Jan.  19,  2012). 

i  n 

Sponsors  are  generally  the  military  services,  but  can  also  be  other  DOD  agencies  or 
combatant  commands. 
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of  gaps,  and  if  a  materiel  solution  is  required,  helps  inform  the  initial 
stages  of  the  acquisition  process,  which  include  an  analysis  of  the 
alternative  solutions  to  fulfilling  the  capability  need  and  the  selection  of  a 
preferred  system  concept.  When  the  technology  development  phase  of 
the  acquisition  process  is  complete,  a  program  sponsor  completes  a 
capability  development  document  that  includes  more  detail  on  the  desired 
capabilities  of  the  proposed  system  and  defines  the  system’s  key 
performance  parameters  or  attributes  against  which  the  delivered 
increment  of  capability  will  be  measured.  Finally,  the  sponsor  prepares  a 
capability  production  document  to  describe  the  actual  performance  of  the 
system  that  will  deliver  the  required  capability.  Figure  1  depicts  how 
JCIDS  reviews  align  with  the  acquisition  process. 


Figure  1:  Interaction  between  DOD’s  Requirements  Generation  and  Acquisition 
Processes 


Initial  Analysis  of  Capability  Capability 

capabilities  document  alternatives*  development  document  production  document 


Capabilities  Analysis  of 

assessment  alternatives 


Technology 

development 


Engineering  and 
manufacturing 
development 


Production 


Source:  GAO  analysis  of  DOD  policy. 


aThe  2012  JCIDS  guidance  directs  the  Functional  Capabilities  Boards  to  review  the  analysis  of 
alternatives  and  other  analyses  for  certain  capability  requirements.  See  Chairman  of  the  Joint  Chiefs 
of  Staff  Instruction  31 70.01  H,  Joint  Capabilities  Integration  and  Development  System,  enc.  A,  para. 
2(c)(1)(b)  (Jan.  10,  2012)  (hereinafter  cited  as  CJCSI  31 70.01  H  (Jan.  10,  2012)). 


The  House  Armed  Services  Committee  and  a  panel  established  by  the 
committee  have  discussed  long-standing  challenges  with  the  JCIDS 
process  and  the  JROC’s  fulfillment  of  its  statutory  responsibilities.  The 
House  Armed  Services  Committee,  in  a  report  accompanying  a  bill  for  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2008,  described  a 
legislative  provision  that  would  allow  for  joint  decision  making  as  opposed 
to  service-centric  budget  considerations  by  incorporating  clear  priorities 
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and  budget  guidance  into  the  JROC  process.11  In  2009,  the  House  Armed 
Services  Committee  established  a  panel  on  defense  acquisition  reform 
because  of  a  sense  that  the  acquisition  system  was  not  responsive 
enough  for  today’s  needs,  not  rigorous  enough  in  protecting  taxpayers, 
and  not  disciplined  enough  in  the  acquisition  of  weapon  systems  for 
tomorrow’s  wars.  The  panel  received  testimony  that  the  Joint  Staff  lacked 
some  of  the  analytical  expertise  necessary  to  ensure  that  the  JCIDS 
process  rigorously  vets  proposed  requirements.  Additionally,  since  2008 
we  have  reported  on  these  challenges.  We  reported  in  2008  that  the 
JCIDS  process  was  not  effective  in  prioritizing  capability  gaps,  and  we 
noted  that  capability  needs  continued  to  be  proposed  and  defined  by  the 
services  with  little  involvement  from  the  joint  community.  We 
recommended  that  the  Secretary  of  Defense  direct  the  Chairman  of  the 
Joint  Chiefs  of  Staff  to  develop  an  analytic  approach  within  JCIDS  to 
better  prioritize  and  balance  the  capability  needs  of  the  military  services, 
combatant  commands,  and  other  defense  components.12  DOD  partially 
agreed  with  our  recommendation  but  did  not  fully  implement  it,  and 
prioritization  remains  service  driven.  More  recently,  in  June  2011  we 
reported  that  the  JROC  did  not  always  consider  trade-offs  among  cost, 
schedule,  and  performance  objectives;  prioritize  requirements;  consider 
redundancies  across  proposed  programs;  or  prioritize  and  analyze 
capability  gaps  in  a  consistent  manner.13  We  recommended  that  the 
JROC  require  higher-quality  resource  estimates  from  requirements 
sponsors  to  ensure  that  best  practices  are  being  followed,  provide  a 
sound  basis  to  ensure  that  trade-offs  are  considered,  prioritize 
requirements  across  proposed  programs,  and  address  potential 
redundancies  during  requirements  reviews,  among  other  steps.  DOD 
partially  agreed  with  our  recommendations  and  commented  that 
improvements  to  the  quality  of  resource  estimates  would  be  addressed  in 
upcoming  changes  to  the  JCIDS  process. 


^See  H.R.  Rep.  No.  110-146,  at  381  (2007)  (accompanying  H.R.  1585).  The  provision, 
which  revised  parts  of  section  181  of  Title  10,  U.S.  Code,  was  included  as  amended  in  the 
National  Defense  Authorization  Act  for  Fiscal  Year  2008.  See  Pub.  L.  No.  110-181,  §  942 
(2008). 

12GAO,  Defense  Acquisitions:  DOD’s  Requirements  Determination  Process  Has  Not  Been 
Effective  in  Prioritizing  Joint  Capabilities,  GAO-08-1060  (Washington,  D.C.:  Sept.  25, 
2008). 

13GAO,  DOD  Weapon  Systems:  Missed  Trade-off  Opportunities  During  Requirements 
Reviews,  GAO-11-502  (Washington,  D.C.:  June  16,  2011). 
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In  May  201 1 ,  we  also  reported  that  combatant  command14  officials  raised 
concerns  that  JCIDS  focuses  more  on  long-term  service-centric  capability 
gaps  than  on  combatant  commands’  more  immediate  and  largely  joint 
gaps.15  JCIDS  was  designed  as  a  deliberate  process  to  meet  longer-term 
joint  needs.  To  address  urgent  needs,  DOD  established  a  separate 
process — the  joint  urgent  operational  needs  process — in  2005.  The  joint 
urgent  operational  needs  process  was  intended  to  respond  to  needs 
associated  with  combat  operations  in  Afghanistan  and  Iraq  and  the  War 
on  Terror.  The  revised  JCIDS  guidance  canceled  separate  guidance  for 
joint  urgent  operational  needs  and  incorporates  and  describes  the  joint 
urgent  operational  needs  process.  Urgent  operational  needs,  as  defined 
by  the  new  JCIDS  guidance,  are  capability  requirements  needed  for 
ongoing  or  anticipated  contingency  operations  that  if  left  unfulfilled  could 
potentially  result  in  loss  of  life  or  critical  mission  failure.16  For  this  report, 
we  focus  on  requirements  that  have  not  been  identified  as  urgent  and 
instead  follow  the  deliberative  JCIDS  process. 


The  Joint  Staff  Has 
Initiated  Efforts  to 
Prioritize  Capability 
Needs,  but 
Implementation 
Processes  Are  Not 
Fully  Developed  and 
Clearly  Documented 


The  Joint  Staff  is  undertaking  efforts  to  improve  the  ability  to  prioritize 
capability  needs  from  a  joint  perspective  through  JCIDS  and  to  align 
those  needs  with  available  budget  resources.  However,  implementation 
processes  for  JCIDS’s  new  approach  to  managing  requirements  and 
considering  affordability  are  still  evolving  and  have  not  been  fully 
developed  and  clearly  documented. 


14Combatant  commands  are  DOD’s  operational  commands.  Of  the  nine  combatant 
commands,  the  following  six  have  geographic  responsibilities:  U.S.  Africa  Command,  U.S. 
Central  Command,  U.S.  European  Command,  U.S.  Northern  Command,  U.S.  Pacific 
Command,  and  U.S.  Southern  Command.  The  following  three  have  functional 
responsibilities:  U.S.  Special  Operations  Command,  U.S.  Strategic  Command,  and  U.S. 
Transportation  Command. 

15GAO,  Defense  Management:  Perspectives  on  the  Involvement  of  Combatant 
Commands  in  the  Development  of  Joint  Requirements,  GAO-1 1-527R  (Washington,  D.C.: 
May  20,  2011). 

16See  CJCSI  3170.01H,  at  GL-7  (Jan.  19,  2012). 
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Changes  in  JCIDS 
Processes  Are  Intended  to 
Improve  Prioritization  of 
Capability  Needs  and 
Guide  Future  Investments 


Determining  priorities  among  joint  requirements  has  been  a  responsibility 
of  the  JROC  since  Congress  amended  section  181  of  Title  10  of  the  U.S. 
Code  in  2008  to  require  the  JROC  to  assist  in  establishing  and  assigning 
priority  levels  for  joint  military  requirements  and  to  help  ensure  that 
resource  levels  associated  with  those  requirements  are  consistent  with 
the  level  of  priority.17  DOD  officials  acknowledge  that  JCIDS  has  been 
ineffective  in  helping  the  JROC  carry  out  these  responsibilities.  We  have 
previously  reported  that  JCIDS’s  ability  to  align  resources  to  balance 
competing  needs  could  be  improved  if  it  had  an  analytic  approach  that 
provided  a  means  to  review  and  validate  proposals  to  ensure  that  the 
most  important  capability  needs  of  the  department  are  being  addressed. 
We  further  said  that  such  an  approach  should  establish  criteria  and 
measures  for  identifying  capability  gaps  and  determining  the  relative 
importance  of  capability  needs.  Finally,  the  approach  should  result  in 
measurable  progress  in  allocating  resources  in  order  to  eliminate 
redundancies,  gain  efficiencies,  and  achieve  a  balanced  mix  of 
executable  programs.18 


DOD  officials  told  us  that  downward  pressure  on  the  defense  budget  has 
led  the  Joint  Staff  to  change  how  the  JCIDS  process  is  used  to  strengthen 
its  ability  to  support  JROC  members  in  making  trade-off  decisions  among 
requirements  and  balancing  risks  across  the  force  within  expected 
resources.  In  fall  2011,  according  to  officials,  the  incoming  Vice  Chairman 
of  the  Joint  Chiefs  of  Staff,  as  the  Chairman  of  the  JROC,  began  to  make 
changes  in  the  JCIDS  processes  to  focus  on  what  capabilities  currently 
exist  and  weigh  the  benefits  of  investing  in  new  capabilities  with  their 
estimated  costs  early  in  the  review  process.  The  Joint  Staff  issued  draft 
guidance  in  October  201 1  and  began  implementation  based  on  the  draft 
guidance.  The  Joint  Staff  issued  the  final  guidance  in  January  2012. 
Descriptions  of  these  changes  and  their  implementation  follow: 

•  New  capabilities  will  be  considered  as  part  of  a  “capability 
portfolio  approach.”  Under  the  portfolio  approach,  officials  stated 
that  JROC  members  are  to  ensure  that  proposed  investments  in 
capabilities  address  joint  needs  or  they  will  not  be  validated  to 


17See  Pub.  L.  No.  110-181,  §  942(a)  (codified  as  amended  at  10  U.S.C.  §  181(b)). 

1 ft 

GAO,  Best  Practices:  An  Integrated  Portfolio  Management  Approach  to  Weapon  System 
Investments  Could  Improve  DOD’s  Acquisition  Outcomes,  GAO-07-388  (Washington, 

D.C.:  Mar.  30,  2007). 
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proceed  to  the  acquisition  process.  In  addition  to  validating  capability 
proposals,  according  to  Joint  Staff  officials,  the  JROC  has  begun 
examining  how  the  services’  existing  programs  can  support  joint 
operations  to  reduce  duplication  of  capabilities.  As  of  December  201 1 , 
according  to  Joint  Staff  officials,  the  last  five  closed  meetings  of  the 
JROC  began  conversations  about  how  to  meet  requirements  by 
considering  available  capabilities  and  the  costs  and  benefits  of 
proposed  programs.  As  a  result,  according  to  Joint  Staff  officials,  at 
least  five  classified  programs  have  been  reviewed  and  altered  by 
either  comparing  redundant  capabilities,  reducing  capacity,  adjusting 
delivery  schedules,  or  directing  follow-on  analysis  before  moving 
programs  forward.  To  support  JROC  decision  making,  officials 
reported  that  Functional  Capabilities  Boards  will  be  tasked  with 
examining  requirements,  their  associated  capability  gaps,  and 
proposed  solutions  within  their  capability  portfolios,  and  independently 
assessing  how  a  proposed  capability  fits  into  its  corresponding  joint 
capability  area.  The  Functional  Capabilities  Boards  previously  had 
responsibility  for  identifying,  assessing,  and  prioritizing  (if  required) 
joint  capability  needs  proposals  within  assigned  joint  capability  areas 
but,  according  to  officials,  have  not  always  carried  out  these 
responsibilities.  Previously,  Functional  Capabilities  Boards  acted 
primarily  as  technical  reviewers  of  requirements  documents,  and 
program  sponsors  briefed  the  JROC  on  the  attributes  of  the  program. 
However,  the  new  guidance  does  not  specify  how  the  independent 
assessment  is  to  be  conducted,  and  it  is  too  soon  to  tell  how  the 
Functional  Capabilities  Boards  will  respond  to  the  new  requirement. 
Officials  reported  that  Functional  Capabilities  Boards  are  expected  to 
develop  methodologies  on  a  case-by-case  basis.  DOD  officials  said 
that  the  analytic  approach  and  uses  of  analytic  information  will  evolve 
over  time. 

DOD  has  previously  attempted  to  manage  capabilities 
departmentwide  through  a  portfolio  approach,  but  has  never  fully 
implemented  the  approach.  In  2006,  DOD  established  an  effort  to 
manage  resources  across  capability  areas  by  establishing  capability 
portfolio  managers  to  enable  DOD  to  develop  and  manage 
capabilities  across  the  department  rather  than  by  military  service  or 
individual  program,  and  by  doing  so,  to  improve  the  interoperability  of 
future  capabilities,  minimize  capability  redundancies  and  gaps,  and 
maximize  capability  effectiveness.  However,  as  we  reported  in  2008, 
capability  portfolio  managers  make  recommendations  on  capability 
development  issues  within  their  portfolios  but  do  not  have 
independent  decision-making  authority.  In  201 1 ,  Joint  Staff  officials 
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told  us  that  they  were  unaware  of  capability  portfolio  managers’  active 
involvement  in  the  JCIDS  process. 

•  Attendance  at  JROC  meetings  will  be  limited  to  key  decision 
makers  and  stakeholders.  Beginning  in  October  2011,  the  JROC 
Chairman  began  to  limit  attendance  at  JROC  meetings  to  facilitate 
candid  discussion  among  senior  leaders  about  priorities  for  joint 
requirements  and  alternative  solutions.  Joint  Staff  officials  told  us  that 
previously,  meetings  were  open  to  a  broad  range  of  interested  parties 
and  service  sponsors  provided  briefings  on  their  proposals  for  new 
capabilities.  Under  the  new  approach  being  implemented,  a 
representative  of  one  of  the  Functional  Capabilities  Boards  will 
provide  a  briefing  on  the  proposal  to  the  JROC.  The  representatives 
would  then  present  the  board’s  independent  assessment  of  benefits, 
costs,  and  risk  for  the  JROC  to  discuss  and  decide  upon.  According  to 
JCIDS  officials,  JROC  members  are  expected  to  make  decisions  from 
the  perspective  of  the  joint  force  and  avoid  taking  a  service-centric 
approach. 

•  Affordability  of  proposals  will  be  a  primary  factor  in  validation 
decisions.  JROC  members  have  always  been  expected  to  consider 
the  resource  implications  of  validation  decisions,  but  officials  stated 
that  until  recently,  these  considerations  have  not  been  a  focus 
because  capabilities  were  not  competing  with  each  other  for  funding. 
According  to  officials,  the  Functional  Capabilities  Boards  have  been 
directed  to  take  similar  steps  to  ensure  that  capability  proposals  not 
only  meet  technical  requirements  but  also  represent  the  most  efficient 
alternative  for  providing  a  capability  within  the  joint  capability  area 
without  creating  duplication  or  overlap.  According  to  JCIDS  officials, 
the  JROC  is  also  reconsidering  previous  validation  decisions  and 
asking  for  changes  to  proposals  to  minimize  costs.19  We  reviewed 
recent  Functional  Capabilities  Board  briefings  to  the  JROC,  which 
provided  information  on  how  needs  might  be  met  with  current 
capabilities  and  alternatives  that  might  meet  needs  while  minimizing 
costs.  However,  it  is  too  soon  to  assess  how  the  JROC  will  consider 
affordability  of  programs  when  making  validation  decisions. 


1 Q 

We  are  not  including  specific  examples  of  issues  and  programs  under  review  because  of 
the  sensitivity  of  deliberations  and  security  classification  concerns. 
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New  JCIDS  Processes  Are 
Not  Fully  Developed  and 
Clearly  Documented 


The  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  has  sought  some  policy 
changes  and,  according  to  officials,  provided  other  direction  to  implement 
changes  in  the  JCIDS  process,  but  the  new  approach  to  managing 
requirements  and  considering  affordability  is  still  evolving  and  has  not 
been  fully  developed  and  clearly  documented.  According  to  Joint  Staff 
officials,  the  dynamic  fiscal  environment  and  the  evolutionary  method 
being  used  to  develop  the  new  approach  and  implementation  processes 
make  it  important  that  decision  makers  maintain  flexibility  in  decision 
making.  We  believe  that  the  new  approach  has  promise  in  positioning  the 
JROC  to  more  accurately  identify  capability  gaps  and  trade-offs,  but  it  has 
not  been  fully  developed  to  include  steps  to  ensure  that  the  approach  is 
fully  implemented,  that  the  intent  is  fully  communicated  to  all  stakeholders 
involved,  and  that  the  results  of  the  new  approach  will  be  measurable. 

We  have  previously  reported  that  key  practices  for  results-oriented 
management  involve  leadership  from  top  officials  as  well  as  the 
involvement  of  stakeholders  at  all  levels  throughout  a  period  of 
transition.20  We  have  also  reported  that  in  order  to  demonstrate  a 
successful  results-oriented  framework,  officials  must  include  clearly 
defined  measures  to  assess  intended  outcomes.21  As  shown  in  table  1, 
key  practices  for  supporting  change  should  include,  among  other  actions, 
obtaining  and  sustaining  support  from  senior  leadership  to  facilitate  the 
transformation,  establishing  clear  lines  of  communication  between  all 
affected  parties,  and  demonstrating  value  and  credibility  of  new 
processes  through  the  use  of  metrics. 


GAO,  Results-Oriented  Cultures:  Implementation  Steps  to  Assist  Mergers  and 
Organizational  Transformations ,  GAO-03-669  (Washington,  D.C.:  July  2,  2003). 

21GAO,  Defense  Management:  Tools  for  Measuring  and  Managing  Defense  Agency 
Performance  Could  Be  Strengthened,  GAO-04-919  (Washington,  D.C.:  Sept.  13,  2004). 
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Table  1:  Elements  of  a  Results-Oriented  Management  Framework 


Principle 


Critical  elements 


Ensure  top  leadership  drives  •  Define  and  articulate  a  succinct  and  compelling 
the  transformation.  reason  for  change. 

Balance  continued  delivery  of  services  with 
transformation  activities. 

Communicate  early  and  often  to  build  trust. 

Ensure  consistency  of  message. 

Encourage  two-way  communication. 

Provide  information  to  meet  specific  needs  of 
employees. 

Each  of  the  agency’s  strategic  plan  goals  is  supported 
by  performance  measures. 

Measures  align  with  departmentwide  goals. 

Goals  and  measures  cascade  from  the  corporate 
level  to  the  lowest  level  of  the  agency. 

Demonstrate  results.  •  Includes  a  combination  of  output-  and  outcome- 

oriented  measures. 

•  Measures  are  clearly  defined. 

•  Provides  trend  data  to  demonstrate  progress  against 
targeted  performance. 


Align  goals  and  measures 
with  agency  and 
departmentwide  goals. 


Establish  a  communication 
strategy  to  create  shared 
expectations  and  report 
related  progress. 


Source:  GAO. 


According  to  Joint  Staff  officials,  the  Chairman  of  the  JROC  has  been  a 
driving  force  in  positioning  the  JROC  to  take  on  the  responsibility  of 
aligning  needs  and  balancing  risk  and  resources,  which  fulfills  one  of  the 
key  steps  for  results-oriented  management,  but  the  approach  is  still  new 
and  officials  have  not  completed  all  of  the  steps  that  facilitate  institutional 
acceptance  and  implementation  of  the  new  approach.  The  Joint  Staff  has 
begun  the  process  of  change  by  articulating  a  clear  rationale  for 
change — that  the  JROC  can  more  effectively  represent  the  warfighters’ 
requirements  and  make  strategic  trade-off  decisions  as  budgets  stay  flat 
or  decrease  by  taking  a  more  active  role  in  shaping  an  affordable  joint 
force.  However,  best  practices  for  managing  a  results-oriented  change 
state  that  goals  and  procedures  should  be  communicated  to  stakeholders 
throughout  the  organization  so  that  they  understand  how  they  should 
implement  the  new  approach  and  how  the  organization  will  measure 
progress.  The  Joint  Staff  issued  guidance  that  outlines  new  procedures 
intended  to  establish  an  approach  for  prioritizing  capabilities  from  a  joint 
perspective  and  to  increase  the  timeliness  of  JCIDS  reviews  by 
categorizing  proposals  according  to  the  level  of  urgency  of  the  need  and 
streamlining  procedures  for  urgent  needs.  However,  the  guidance  does 
not  clearly  outline  criteria  and  measures  for  demonstrating  progress 
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toward  meeting  the  goal  of  aligning  needs  with  available  resources  or 
clearly  communicate  the  goals  and  the  analytic  approach  envisioned  to 
support  JROC  decision  making.  Further,  the  guidance  does  not  describe 
how  the  proposed  change  will  affect  the  services,  combatant  commands, 
and  other  stakeholders.  Finally,  the  JCIDS  guidance  does  not  establish 
criteria  and  measures  for  demonstrating  progress  toward  the  goal  of 
creating  a  balanced  portfolio  of  programs  that  takes  into  account  needs, 
risks,  and  available  resources,  nor  do  other  documents  provided  to  us  by 
DOD.  Measures  such  as  the  proportion  of  requirements  that  address  joint 
priority  needs  versus  service-specific  needs,  the  savings  obtained 
through  the  elimination  of  redundant  capabilities,  and  the  comparison  of 
estimated  costs  of  a  proposed  program  with  the  actual  costs  of  operating 
the  program  over  its  life  cycle  could  be  helpful  in  assessing  whether  the 
process  is  balancing  requirements  with  available  resources  or  whether 
further  adjustments  to  the  process  are  needed. 

Considering  new  capabilities  across  the  department  in  the  context  of  joint 
capability  areas  can  help  DOD  begin  to  identify  priorities  for  future 
investment.  However,  unless  the  Joint  Staff  takes  steps  to  define  and 
institutionalize  the  new  approach  by  adhering  to  the  key  principles  of 
results-oriented  management,  it  is  not  clear  whether  the  current 
momentum  in  implementing  an  analytic  approach  through  JCIDS  will  be 
sustained. 


JROC  Has  Not  Always 
Had  Complete 
Information  on  Future 
Costs  to  Sustain 
Proposed  Capabilities 
before  Approving 
Documentation  for 
Program 
Development 


Even  though  sustainment  costs  make  up  a  significant  portion  of  the  total 
ownership  costs  of  a  weapon  system,  the  JROC  has  not  always  had 
complete  information  on  such  costs  when  validating  documentation  used 
in  the  decision  to  initiate  program  development.  “Sustainment”  as  a 
category  represents  a  range  of  activities  intended  to  ensure  the 
availability  of  weapon  systems  and  to  support  their  operations,  including 
some  logistics  and  personnel  services.  During  the  identification  of 
capability  gaps  and  consideration  of  selected  alternatives,  it  is  difficult  for 
the  sponsors  to  provide  detailed  information  on  program  capabilities  and 
cost  estimates.  As  a  major  defense  acquisition  program  moves  toward 
the  development  stage,  JCIDS  requires  that  more  complete  and  accurate 
sustainment  information  be  presented  in  capability  development 
documents.  According  to  DOD  officials,  decision  makers  need  more 
accurate  cost  information  to  assess  whether  the  benefit  of  a  proposed 


Page  14 


GAO-1 2-339  Defense  Management 


capability  justifies  the  cost  of  sustaining  the  capability  over  its  life.22 
JCIDS  guidance  requires  that  sponsors  of  potential  major  defense 
acquisition  programs23  include  sustainment  information  in  capability 
development  documents,24  which  detail  proposed  solutions  to  fulfill 
capability  needs. 

The  JROC  has  generally  relied  on  sponsor-provided  assessments  of 
sustainment  information  in  the  capability  development  documents  to 
make  its  validation  decisions,  but  these  documents  have  not  always 
included  all  the  information  suggested  in  JCIDS  guidance  or  sufficient 
detail  to  enable  the  JROC  to  assess  the  quality  of  the  information.  A  DOD 
manual25  regarding  the  development  of  sustainment  information  suggests 
that  when  sustainment  requirements  and  underlying  assumptions  are  not 
clearly  documented,  subsequent  decisions  about  the  project  may  be 
based  on  incorrect  assumptions.  Prior  GAO  work26  suggests  that  gaps  in 
joint  warfighting  capabilities  and  proposals  to  fulfill  the  gaps  should  be 
clearly  identified  to  decision-making  bodies,  such  as  the  JROC,  to  inform 
deliberations.  Further,  information  should  be  complete  so  those  making 
the  important  decisions  may  do  so  as  effectively  as  possible. 


220ur  prior  work  has  found  that  the  operation  and  support  costs  of  sustaining  a  fielded 
system  account  for  about  70  percent  of  a  system’s  life  cycle  costs.  See  GAO-10-717. 

oo 

During  our  review,  the  Joint  Staff  revised  guidance  on  sustainment  reporting 
requirements.  Under  the  guidance  in  effect  through  December  201 1 ,  which  we  used  for 
our  review,  sustainment  metrics  were  to  be  developed  for  all  acquisition  category  I 
programs  involving  materiel  solutions  and  acquisition  category  II  and  below  programs  as 
determined  by  the  sponsor.  See  Manual  for  the  Operation  of  the  Joint  Capabilities 
Integration  and  Development  System,  app.  B  to  enc.  B,  para.  2  (February  2009,  updated 
Jan.  31 , 201 1 )  (hereinafter  cited  as  JCIDS  Manual  (Jan.  31 , 201 1 )).  According  to  the 
recently  revised  JCIDS  Manual,  sustainment  metrics  must  be  developed  for  all  acquisition 
category  I  programs.  The  manual  also  states  that  acquisition  category  II  and  below 
programs,  with  material  solutions,  shall  include  the  sustainment  metrics  or  sponsor- 
defined  sustainment  metrics.  See  JCIDS  Manual,  app.  E  to  enc.  B,  para.  2  (Jan.  19, 
2012).  Under  both  versions  of  the  manual,  these  metrics  are  developed  in  the  form  of  key 
performance  parameters  and  key  system  attributes. 

24The  JCIDS  Manual  required  the  inclusion  of  sustainment  information  in  capability 
development  documents  beginning  in  2007. 

25DOD,  Reliability,  Availability,  Maintainability,  and  Cost  Rationale  Report  Manual 
(Washington,  D.C.:  June  1, 2009). 

26GAO,  Defense  Acquisition:  Employing  Best  Practices  Can  Shape  Better  Weapon 
System  Decisions,  GAO/T-NSIAD-OO-137  (Washington,  D.C.:  Apr.  26,  2000). 
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JCIDS  guidance  requires  sponsors  of  major  defense  acquisition  programs 
to  address  sustainment  based  on  four  metrics — materiel  availability, 
operational  availability,  reliability,  and  ownership  cost  (renamed  operation 
and  support  cost  in  January  2012). 27  The  guidance  includes  a  series  of 
criteria,  which  are  listed  as  review  criteria  in  the  guidance.  The  criteria 
provide  additional  information  on  each  metric.  For  example,  JCIDS 
guidance  lists  as  review  criteria  for  the  materiel  availability  metric  whether 
there  is  a  clear  definition  and  accounting  for  the  intended  service  life  of 
the  program,  an  identification  of  planned  downtime,  and  a  comparison  of 
downtime  value  with  experiences  of  other  analogous  systems,  among 
other  criteria.  Table  2  outlines  examples  of  key  review  criteria  within  each 
of  the  four  sustainment  metrics  under  the  guidance  in  effect  through 
December  201 1 ,28 


27 See  JCIDS  Manual,  app.  B  to  enc.  B  (Jan.  31 , 201 1 ).  See  also  JCIDS  Manual,  app.  E  to 
enc.  B  (Jan.  19.  2012).  The  version  of  the  JCIDS  Manual  in  effect  through  December  201 1 
defined  “materiel  availability”  as  a  measure  of  the  percentage  of  the  total  inventory  of  a 
system  that  is  operationally  capable  (ready  for  tasking)  of  performing  an  assigned  mission 
at  a  given  time,  based  on  materiel  condition;  “operational  availability”  as  a  measure  of  the 
percentage  of  time  that  a  system  or  group  of  systems  within  a  unit  are  operationally 
capable  of  performing  an  assigned  mission  and  can  be  expressed  as  (uptime/(uptime  + 
downtime));  “reliability”  as  a  measure  of  the  probability  that  the  system  will  perform  without 
failure  over  a  specific  interval  and  must  be  sufficient  to  support  the  warfighting  capability 
needed;  and  “ownership  cost”  as  a  measure  that  provides  balance  to  the  sustainment 
solution  by  ensuring  that  the  operations  and  support  costs  associated  with  availability  are 
considered  in  making  decisions,  and  should  cover  the  planned  lifecycle  timeframe, 
consistent  with  the  timeframe  used  in  the  materiel  availability  metric.  The  revised  guidance 
largely  maintained  these  descriptions. 

no 

The  revised  guidance  issued  in  January  2012  includes  many  of  the  same  review  criteria. 
See  JCIDS  Manual,  app.  E  to  enc.  B  (Jan.  19,  2012).  As  previously  noted,  the  revised 
guidance  renamed  the  ownership  cost  metric  as  operation  and  support  cost. 
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Table  2:  Examples  of  Review  Criteria  for  Each  of  the  Four  Sustainment  Metrics 


Sustainment 

metric 

Total  number  of 
review  criteria 

Examples  of  review  criteria 

Materiel 

availability 

8 

Accounts  for  the  total  population  of  end  items 
being  acquired  for  operational  use 

Identifies  and  includes  planned  time  periods 
during  which  the  weapon  system  is  not 
operationally  available  to  perform  an  assigned 
mission 

Identifies  sources  of  data  and  models  used  to 
establish  and  track  materiel  availability 

Operational 

availability 

5 

Provides  specific  definitions  for  what  constitutes 
failure 

Addresses  time  periods  during  which  the 
weapon  system  is  nonfunctional  as  a  result  of 
failure 

Addresses  specific  issues  associated  with 
failure,  including  time  needed  to  recover  a 
weapon  system  after  suffering  a  failure  and 
time  to  perform  diagnostics 

Reliability 

8 

Defines  how  time  intervals  for  assessing  the 
weapon  system  will  be  measured 

Identifies  sources  for  baseline  reliability  data 
and  addresses  whether  the  proposed  reliability 
value  is  consistent  with  comparable  systems 
Addresses  whether  the  proposed  reliability  of 
the  weapon  system  is  consistent  with  intended 
operational  use  of  system 

Ownership 

cost 

9 

Defines  through  analysis  which  type  of 
ownership  cost  structure  was  used  for 
assessing  program  cost 

Includes  all  required  costs  for  a  program, 
regardless  of  funding  source 

Includes  an  approach  to  monitor,  collect,  and 
validate  operation  and  support  cost  data 

Source:  GAO  analysis  of  DOD  guidance. 


Program  sponsors  provide  initial  information  on  the  sustainment  metrics 
for  proposed  capability  solutions  when  they  submit  a  capability 
development  document,  one  of  three  capability  documents  the  JROC 
considers  in  its  review  and  potential  validation  of  capability  proposals. 
Officials  from  both  the  Joint  Staff  and  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  review  the  capability 
development  documents  and,  according  to  officials,  verify  that  all  required 
sustainment  elements  have  been  included  before  the  documents  are 
validated  by  the  JROC.  Officials  also  provide  their  independent 
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assessments  of  the  quality  of  the  cost  estimates  to  the  JROC.29  Officials 
from  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  said  they  work  with  program  sponsors  to 
ensure  that  life  cycle  sustainment  planning  and  costs  are  as  accurate  as 
possible.  Officials  from  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  and  the  Joint  Staff  told  us  that  they 
consider  reported  sustainment  information  important  to  a  program’s 
development  and  review  all  reported  information. 

The  JCIDS  Manual  notes  that  listed  criteria,  information,  and  activities 
cannot  necessarily  be  applied  to  all  systems.  Sponsors  have  a  degree  of 
latitude  in  determining  which  items  are  applicable  for  their  specific 
concept,  technology,  system,  or  combination  of  these.  For  example,  a 
program  sponsor  for  a  major  defense  acquisition  program  is  required  to 
report  a  measure  for  operational  availability,  but  would  not  necessarily 
have  to  report  on  the  respective  criteria,  such  as  addressing  downtime 
associated  with  failure,  including  recovery  time  or  movement  of 
maintenance  teams  to  the  work  site.  Because  the  guidance  does  not 
specifically  require  program  sponsors  to  report  on  the  individual  criteria, 
they  generally  include  some,  but  not  all,  of  the  individual  criteria. 

Our  analysis  of  six  capability  development  documents  found  that  all  of  the 
documents  provided  information  on  all  of  the  required  sustainment 
metrics.  However,  we  found  that  the  completeness  of  information 
reported  for  all  of  the  metrics’  key  criteria  varied.  Specifically,  none  of  the 
documents  included  complete  information  for  each  of  the  four 
sustainment  metrics’  review  criteria  elements.  In  addition,  each  of  the 
documents  had  some  common  omissions;  for  example,  none  of  the  six 
capability  documents  we  reviewed  included  information  on  all  nine  of  the 
ownership  cost  metric’s  criteria  elements.  Further,  several  of  these 
documents  only  included  information  on  one  criteria  element  for  a  single 
metric,  and  none  reported  information  on  all  of  the  elements  for  any  of  the 
metrics.  Finally,  when  information  on  the  metrics’  key  criteria  was 
provided,  the  level  of  detail  varied  among  the  documents.  For  example, 
for  some  criteria,  some  documents  provided  a  paragraph  of  supporting 
information  and  analysis  whereas  others  provided  single-sentence 


pq 

The  Joint  Staff  Directorate  for  Logistics  (J4),  Maintenance  Division,  is  the  Joint  Staff 
entity  responsible  for  evaluating  the  sustainment  key  performance  parameter.  In  this 
report,  GAO  refers  to  the  sustainment  key  performance  parameter  as  sustainment 
information. 
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responses  that  did  not  provide  as  substantial  detail  for  the  supporting 
criteria.  According  to  officials,  for  the  capability  documents  we  reviewed, 
there  were  cases  in  which  information  on  some  of  the  suggested  criteria 
should  have  been  included  but  were  not. 

Joint  Staff  officials  and  officials  from  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics  noted  that  while  criteria 
for  each  of  the  required  sustainment  metrics  may  not  be  applicable  to 
every  program,  it  would  be  beneficial  to  the  JROC  if  the  services  reported 
on  the  criteria  for  each  metric  outlined  in  the  guidance  or  indicated  a 
reason  why  a  specific  criterion  was  not  applicable.  Officials  from  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics  told  us  that  when  they  conduct  their  reviews  of  sponsor-reported 
sustainment  information,  not  all  of  the  supporting  documentation  they 
need  for  a  thorough  independent  assessment  is  available  in  the  capability 
development  document.  These  officials  said  that  they  can  generally  find 
detailed  documentation  on  sustainment  planning  and  costs  from  sources 
outside  of  the  JCIDS  process,  but  that  the  information  is  not  always 
readily  available  within  the  JCIDS  database.  Ultimately,  review  efficiency 
could  be  improved  if  all  the  information  were  available  in  the  JCIDS 
database.  Updated  JCIDS  guidance  issued  in  January  2012  still  does  not 
clearly  require  program  sponsors  to  report  on  the  individual  criteria  for 
each  of  the  sustainment  metrics.  The  Joint  Staff  is  developing  a  new 
reporting  tool  intended  to  provide  a  standard  format  for  reporting 
sustainment  information.  The  tool  will  require  program  sponsors  to  at 
least  minimally  address  each  of  the  four  sustainment  metrics  in  order  to 
submit  the  capability  development  document  for  review  through  JCIDS, 
according  to  officials.  However,  officials  stated  that  this  tool  will  not 
require  that  sponsors  address  each  of  the  individual  criteria  elements 
within  the  four  sustainment  metrics. 

Without  complete  and  detailed  information  on  each  of  the  individual 
criteria  elements,  the  JROC  may  not  be  in  the  best  position  to  weigh  the 
costs  and  benefits  of  a  proposal  within  a  capability  portfolio.  The  quality 
and  completeness  of  the  data  that  sponsors  provide  through  the 
capability  development  document  in  the  JCIDS  process  will  become  more 
important  as  the  JROC  increases  its  examination  of  the  benefits  of 
programs  balanced  against  their  associated  costs.  As  we  have  previously 
reported,  incomplete  and  inaccurate  sustainment  information  has  been  a 


Page  19 


GAO-12-339  Defense  Management 


long-standing  problem  for  DOD.30  In  November  2009,  a  DOD  team 
assessing  weapon  system  acquisition  reform  reported  that  DOD  lacked 
valid,  measurable  sustainment  information  to  accurately  assess  how 
programmatic  decisions  affected  life  cycle  costs  and  made 
recommendations  to  improve  weapon  system  life  cycle  sustainment.31 

Until  the  JROC  requires  program  sponsors  to  report  complete 
sustainment  information,  including  both  the  overall  metrics  and  the 
supporting  criteria,  the  JROC  may  not  always  have  the  complete  and 
detailed  information  it  needs  to  make  the  most  informed  decisions. 

Conclusions 

The  prospect  of  declining  budgets  has  amplified  the  need  for  DOD  to 
prioritize  among  capability  gaps  and  to  use  its  resources  to  maximize  the 
capabilities  of  the  joint  force.  The  Chairman  of  the  JROC  has  begun  to 
take  steps  to  better  balance  risks  across  the  joint  force  by  examining 
proposals  for  new  capabilities  within  the  context  of  existing  joint  capability 
areas  and  to  consider  affordability,  including  sustainment  costs,  as  a 
factor  in  validating  requirements  proposals.  The  revised  approach  is  new 
and  evolving,  but  in  order  for  it  to  achieve  the  intended  results  of 
prioritizing  capability  needs  and  aligning  those  needs  with  available 
resources,  the  Joint  Staff  needs  to  take  steps  to  fully  develop  the 
approach  and  document  it  more  explicitly.  Specifically,  DOD  does  not  yet 
have  a  documented  implementation  plan  with  measures  of  success  that 
support  change.  In  addition,  having  good  sustainment  information  is  a  key 
element  needed  to  improve  JCIDS’s  success  over  the  long  term. 
Sustainment  costs  historically  represent  70  percent  of  a  system’s  life 
cycle  costs,  but  DOD  has  been  making  decisions  with  incomplete 
information  on  sustainment  and  does  not  require  that  sponsors  address 
all  of  the  criteria  outlined  in  JCIDS  guidance.  Until  the  JROC  has 
developed  and  fully  documented  an  approach  for  prioritizing  capability 
needs  and  aligning  these  needs  with  available  resources  and  has 
complete  sustainment  information  associated  with  the  operation  of  new 
capabilities,  it  will  not  be  in  the  best  position  to  align  resources  with 
priorities  or  balance  costs  with  benefits  in  affordable  investment  plans. 

30GAO-10-717. 

31  DOD,  Weapon  System  Acquisition  Reform  Product  Support  Assessment  (Washington, 
D.C.:  Nov.  1 , 2009).  This  report  was  produced  by  a  DOD  Product  Support  Assessment 
Team  led  by  the  Office  of  the  Deputy  Under  Secretary  of  Defense  for  Logistics  and 

Materiel  Readiness. 
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Recommendations  for 
Executive  Action 

•  Revise  and  implement  guidance  to  reflect  changes  to  the  JCIDS 
process  as  well  as  to  establish  criteria  and  measures  for  determining 
the  relative  importance  of  capability  needs  across  capability  areas  and 
assessing  progress. 

•  Explicitly  require  that  program  sponsors  address  each  of  the  criteria 
outlined  for  the  individual  sustainment  metrics  when  submitting 
capability  development  documents. 


To  help  sustain  momentum  for  efforts  to  bring  a  capability  portfolio 
approach  to  the  JCIDS  process  and  to  improve  the  quality  of  sustainment 
information  reported  in  capability  development  documents,  we 
recommend  that  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff,  as  the 
JROC  Chairman,  take  the  following  two  actions: 


Agency  Comments 
and  Our  Evaluation 


The  Joint  Staff  provided  written  comments  on  a  draft  of  this  report.  In  its 
comments,  the  Joint  Staff  said  our  report  represented  a  fair  and  objective 
assessment  of  the  JCIDS  process.  It  partially  agreed  with  both  of  our 
recommendations,  citing  ongoing  and  planned  changes  to  the  joint 
requirements  development  process.  However,  the  comments  did  not 
detail  any  specific  steps  that  DOD  plans  to  take  to  address  our 
recommendations.  The  comments  are  reprinted  in  their  entirety  in 
appendix  III.  The  Joint  Staff  also  provided  technical  comments,  which  we 
have  incorporated  into  the  report  as  appropriate. 

The  Joint  Staff  partially  concurred  with  our  recommendation  that  the  Vice 
Chairman  of  the  Joint  Chiefs  of  Staff  revise  and  implement  guidance  to 
reflect  changes  to  the  JCIDS  process  as  well  as  to  establish  criteria  and 
measures  for  determining  the  relative  importance  of  capability  needs 
across  capability  areas  and  assessing  progress.  In  its  written  response, 
the  Joint  Staff  described  recent  initiatives  to  substantially  change  the  joint 
requirements  development  process  to  require  that  capability  requirements 
be  evaluated  within  a  capability  portfolio  by  Functional  Capability  Boards, 
the  Joint  Capabilities  Board,  and  the  JROC.  The  Joint  Staff  also 
discussed  its  planned  efforts  to  improve  prioritization  of  capability  needs 
and  stated  that  JROC  reviews  will  incorporate  an  evolving  portfolio 
assessment  tool.  The  Joint  Staff  expects  that  the  departmentwide 
priorities  outlined  in  DOD’s  strategic  guidance  as  well  as  a  revised 
process  for  assessing  capability  gaps  and  combatant  command  priorities 
will  enable  the  JROC  to  make  more  informed  decisions  about  priorities. 
While  we  agree  that  the  Joint  Staff  has  taken  important  steps  to  enable 
prioritization  of  capabilities,  such  as  addressing  prioritization  in  a  new 
enclosure  in  its  revised  JCIDS  Manual ,  the  enclosure  does  not  explicitly 
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outline  implementation  processes.  We  continue  to  believe  that  clear 
guidance  that  establishes  criteria  for  determining  priority  levels  and 
measures  for  demonstrating  progress  will  be  essential  in  sustaining 
momentum  toward  the  goal  of  creating  a  balanced  portfolio  of  programs 
that  takes  into  account  needs,  risks,  and  available  resources.  Moreover, 
providing  guidance  that  fully  documents  the  new  procedures  for  assigning 
priority  levels  to  capability  gaps  is  an  essential  step  toward  clarifying  how 
the  procedures  will  be  implemented. 

The  Joint  Staff  also  partially  concurred  with  our  recommendation  that  the 
Vice  Chairman  of  the  Joint  Chiefs  of  Staff  explicitly  require  that  program 
sponsors  address  each  of  the  criteria  outlined  for  the  individual 
sustainment  metrics  when  submitting  capability  development  documents. 
According  to  its  written  comments,  the  Joint  Staff  criteria  for  the 
sustainment  metrics  were  designed  to  guide  the  development  of 
requirements,  but  were  not  intended  to  be  prescriptive  because  individual 
programs  are  unique  and  criteria  applicable  to  one  problem  may  not  apply 
to  another.  We  agree  that  each  of  the  criteria  may  not  be  applicable  to 
every  program.  However,  if  program  sponsors  addressed  each  criterion  in 
some  manner,  including  explaining  that  a  criterion  is  not  applicable  to  the 
program,  the  JROC  would  be  assured  that  program  sponsors  considered 
all  criteria  when  developing  program  proposals.  Further,  the  Joint  Staff 
commented  that  JCIDS  reviews  of  capability  development  documents  by 
Joint  Staff  and  Office  of  the  Secretary  of  Defense  officials  ensure  that  a 
document  is  thoroughly  vetted  for  consideration  by  the  JROC.  It 
suggested  that  the  inclusion  of  analyses  and  rationale  for  sustainment 
metrics  development  in  capability  development  documents  might  be 
duplicative  because  this  information  is  contained  in  acquisition  documents 
that  exist  outside  of  JCIDS.  However,  as  we  noted  in  our  report,  the 
documents  that  contain  the  analysis  and  rationale  for  the  required 
sustainment  metrics  are  not  necessarily  reviewed  by  or  available  to  the 
JROC  members  during  their  consideration  of  a  capability  development 
document.  We  continue  to  believe  that  the  inclusion  of  a  sponsor- 
provided  rationale  for  each  metric  criterion  would  enhance  the 
thoroughness  and  efficiency  of  the  JROC’s  review  of  sustainment 
information  through  JCIDS. 


We  are  sending  copies  of  this  report  to  interested  congressional 
committees,  the  Secretary  of  Defense,  the  Vice  Chairman  of  the  Joint 
Chiefs  of  Staff,  and  other  interested  parties.  In  addition,  the  report  is 
available  at  no  charge  on  the  GAO  website  at  http://www.gao.gov. 
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If  you  or  your  staff  have  any  questions  about  this  report,  please  contact 
me  at  (202)  512-3489  or  pendletonj@gao.gov.  Contact  points  for  our 
Offices  of  Congressional  Relations  and  Public  Affairs  may  be  found  on 
the  last  page  of  this  report.  GAO  staff  who  made  key  contributions  to  this 
report  are  listed  in  appendix  IV. 


John  Pendleton 

Director,  Defense  Capabilities  and  Management 
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List  of  Committees 


The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

The  Honorable  Daniel  K.  Inouye 
Chairman 

The  Honorable  Thad  Cochran 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
United  States  Senate 

The  Honorable  Howard  P.  “Buck”  McKeon 
Chairman 

The  Honorable  Adam  Smith 
Ranking  Member 
Committee  on  Armed  Services 
House  of  Representatives 

The  Honorable  C.W.  “Bill”  Young 
Chairman 

The  Honorable  Norman  D.  Dicks 
Ranking  Member 
Subcommittee  on  Defense 
Committee  on  Appropriations 
House  of  Representatives 
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Appendix  I:  Observations  on  Reporting 
Provisions  in  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2011 


Section  862  of  the  Ike  Skelton  National  Defense  Authorization  Act  for 
Fiscal  Year  201 1  requires  the  Comptroller  General  to  carry  out  a 
comprehensive  review  of  the  Joint  Capabilities  Integration  and 
Development  System  (JCIDS)  and  to  submit  to  the  congressional  defense 
committees  a  report  on  the  review.1  This  appendix,  in  conjunction  with  the 
letter,  addresses  each  of  the  reporting  provisions  as  described  in  the  act. 
Specifically,  section  862  requires  the  following  contents  for  the  review: 

Purpose.  The  purpose  of  the  review  is  to  evaluate  the  effectiveness  of 
JCIDS  in  achieving  the  following  objectives: 

•  Timeliness  in  delivering  capability  to  the  warfighter. 

•  Efficient  use  of  the  investment  resources  of  the  Department  of 
Defense  (DOD). 

•  Control  of  requirements  creep.2 

•  Responsiveness  to  changes  occurring  after  the  approval  of  a 
requirements  document3  (including  changes  to  the  threat 
environment,  the  emergence  of  new  capabilities,  or  changes  in  the 
resources  estimated  to  procure  or  sustain  a  capability). 

•  Development  of  the  personnel  skills,  capacity,  and  training  needed  for 
an  effective  and  efficient  requirements  process. 

Matters  considered.  In  performing  the  review,  the  Comptroller  General  is 
required  to  gather  information  on  and  consider  the  following  matters: 

•  The  time  that  requirements  documents  take  to  receive  approval 
through  JCIDS. 

•  The  quality  of  cost  information  considered  in  JCIDS  and  the  extent  of 
its  consideration. 


Ree  Pub.  L.  No.  111-383,  §  862(a)  (2011).  The  statute  specifically  defined  JCIDS  by 
reference  to  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  31 70.01  G.  See  §  862(c)(1). 

Requirements  creep  is  defined  in  the  statute  as  “the  addition  of  new  technical  or 
operational  specifications  after  a  requirements  document  is  approved.”  See  §  862(c)(3). 

3The  term  requirements  document  is  defined  in  the  statute  as  “a  document  produced  in 
JCIDS  that  is  provided  for  an  acquisition  program  to  guide  the  subsequent  development, 
production,  and  testing  of  the  program  and  that — (A)  justifies  the  need  for  a  materiel 
approach,  or  an  approach  that  is  a  combination  of  materiel  and  non-materiel,  to  satisfy 
one  or  more  specific  capability  gaps;  (B)  details  the  information  necessary  to  develop  an 
increment  of  militarily  useful,  logistically  supportable,  and  technically  mature  capability, 
including  key  performance  parameters;  or  (C)  identifies  production  attributes  required  fora 
single  increment  of  a  program.”  See  §  862(c)(2). 
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Appendix  I:  Observations  on  Reporting 
Provisions  in  the  National  Defense 
Authorization  Act  for  Fiscal  Year  2011 


•  The  extent  to  which  JCIDS  establishes  a  meaningful  level  of  priority 
for  requirements. 

•  The  extent  to  which  JCIDS  is  considering  trade-offs  between  cost, 
schedule,  and  performance  objectives. 

•  The  quality  of  information  on  sustainment  considered  in  JCIDS  and 
the  extent  to  which  sustainment  information  is  considered. 

•  An  evaluation  of  the  advantages  and  disadvantages  of  designating  a 
commander  of  a  unified  combatant  command  for  each  requirements 
document  for  which  the  Joint  Requirements  Oversight  Council  (JROC) 
is  the  validation  authority  to  provide  a  joint  evaluation  task  force  to 
participate  in  a  materiel  solution  and  to 

•  provide  input  to  the  analysis  of  alternatives;  participate  in  testing 
(including  limited  user  tests  and  prototype  testing);provide  input  on 
a  concept  of  operations  and  doctrine;  provide  end  user  feedback 
to  the  resource  sponsor;  and  participate,  through  the  combatant 
commander  concerned,  in  any  alteration  of  the  requirement  for 
such  solution. 

Section  862  also  provided  definitions  for  JCIDS,  requirements  document, 
requirements  creep,  and  materiel  solution.4 

Tables  3  through  12  contain  our  response  to  each  of  the  requirements 
mandated  by  the  Ike  Skelton  National  Defense  Authorization  Act  for 
Fiscal  Year  201 1 .5  The  provision  mandating  our  report  defined  the  JCIDS 
process  by  referring  to  the  JCIDS  guidance  in  effect  from  March  2009 
through  December  201 1 .6  Accordingly,  our  response  to  the  mandated 
elements  as  presented  in  this  appendix  generally  focuses  on  JCIDS 
under  that  guidance.  However,  we  also  provide  some  information  relating 
to  JCIDS  as  described  in  the  revised  guidance  issued  in  January  2012.  In 
addition,  our  assessments  generally  focused  on  those  programs  that 
were  determined  to  be  JROC-interest,  or  those  that  were  designated  as 
major  defense  acquisition  programs  or  major  automated  information 
systems  and  capabilities  that  have  a  potentially  significant  impact  on 
interoperability  in  allied  and  coalition  operations.  Generally,  these 
programs  have  greater  costs  or  are  more  complex  than  smaller  programs, 


4See  §  862(c). 

5See  Pub.  L.  No.  111-383,  §  862. 

6See  §  862(c)(1)  (defining  JCIDS  as  “the  system  for  the  assessment,  review,  validation, 
and  approval  of  joint  warfighting  requirements  that  is  described  in  Chairman  of  the  Joint 
Chiefs  of  Staff  Instruction  31 70.01  G”). 
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Appendix  1:  Observations  on  Reporting 

Provisions  in  the  National  Defense 

Authorization  Act  for  Fiscal  Year  2011 

and  therefore  provide  an  opportunity  to  assess  the  effectiveness  of  more 
aspects  of  the  JCIDS  process. 

Table  3:  Section  862(b)(1)(A)- 
Warfighter 

—Evaluate  the  Effectiveness  of  JCIDS  in  Achieving  Timeliness  in  Delivering  Capability  to  the 

Summary  assessment 

The  JCIDS  guidance  in  effect  through  December  201 1  did  not  directly  facilitate  the  timely 
delivery  of  capabilities  to  the  warfighter.  Joint  Staff  officials  are  implementing  new 
procedures  to  expedite  reviews  of  urgent  needs  and  establishing  targets  for  anticipated 
delivery  times  for  weapon  systems  to  facilitate  timely  delivery  of  capabilities  to  meet 
needs. 

Key  GAO  observations 

JROC  review  and  validation  of  capability  needs  validated  in  2010  and  2011  ranged  from  3 
months  to  17  months.  Officials  told  us  that  the  length  of  time  it  has  taken  for  a  capability 
need  to  be  validated  through  JCIDS  depended  on  several  factors,  including  the  quality 
and  completeness  of  the  original  proposal  and  the  maturity  of  the  technology  involved. 
According  to  officials,  JCIDS  reviews  generally  did  not  delay  or  expedite  the  delivery  of  a 
capability;  instead,  the  development  phases  of  the  acquisition  system  took  the  most  time 
between  identifying  a  need  and  delivering  a  capability.  JCIDS  guidance  in  effect  through 
December  201 1  did  not  require  the  JROC  to  approve  a  weapon  system’s  anticipated 
schedule  when  making  decisions  about  validating  the  related  requirements  documents, 
and  according  to  Joint  Staff  officials,  the  JROC  typically  has  not  tried  to  influence  the 
delivery  dates  of  capabilities.  Recent  changes  to  JCIDS  processes  require  the  JROC  to 
set  targets  for  the  anticipated  delivery  schedule  when  validating  a  weapon  system’s 
capability  development  document  and  to  revalidate  the  document  if  the  delivery  target  is 
altered  by  more  than  12  months.  In  addition,  JCIDS  now  tailors  requirements  reviews 
according  to  the  urgency  of  the  need.3 

GAO  scope  and  methodology 

To  evaluate  the  extent  to  which  JCIDS  had  contributed  to  the  timely  delivery  of 
capabilities  to  the  warfighter,  we  reviewed  policy  documents  about  the  joint  requirements 
and  acquisition  processes.  Using  the  Joint  Staff’s  database  of  joint  requirements 
documents  reviewed  through  JCIDS,  we  selected  a  nonprobability  sample  of  12 
requirements  documents  for  major  weapon  systems  that  the  JROC  reviewed  from  May  1 , 
2010,  through  May  31 , 201 1 .  We  focused  on  major  weapon  systems  because  they  were 
reviewed  through  each  of  the  JCIDS  phases,  whereas  other  weapon  systems  are 
generally  reviewed  at  the  Joint  Capabilities  Board  level  but  not  by  the  JROC.  Specifically, 
we  analyzed  the  time  that  elapsed  during  the  JCIDS  review  of  four  initial  capability 
documents,  five  capability  development  documents,  and  three  capability  production 
documents.  We  interviewed  DOD,  Joint  Staff,  and  service  officials  with  detailed 
knowledge  of  the  programs  and  the  review  process  to  confirm  our  findings.  Finally,  we 
reviewed  JCIDS  guidance  issued  in  January  2012. 

Sources:  DOD  and  GAO. 


aSee  CJCSI  3170.01H,  enc.  A,  para.  1(d)(2)  (Jan.  10,  2012),  and  JCIDS  Manual,  enc.  E  (Jan.  19, 
2012).  Specifically,  the  guidance  establishes  different  document  staffing  and  validation  processes 
where  a  need  is  a  joint  urgent  operational  need  or  a  joint  emergent  operational  need.  In  2005,  DOD 
established  an  urgent-needs,  expedited  review  process  outside  JCIDS  because  the  JCIDS  process 
was  ineffective  at  addressing  urgent  wartime  needs;  the  revised  JCIDS  instruction  cancels  the 
instruction  driving  that  process.  See  CJCSI  3170. 01H,  para.  2(b)  (Jan.  10,  2012)  (canceling  Chairman 
of  the  Joint  Chiefs  of  Staff  Instruction  3470.01 ,  Rapid  Validation  and  Resourcing  of  Joint  Urgent 
Operational  Needs  (JUONs)  in  the  Year  of  Execution  (July  15,  2005)). 
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Table  4:  Section  862(b)(1)(B) — Evaluate  the  Effectiveness  of  JCIDS  in  Achieving  Efficient  Use  of  the  Investment  Resources  of 
the  Department  of  Defense 


Summary  assessment 

The  JROC  has  not  been  effective  at  prioritizing  capability  needs  or  guiding  the  efficient 
use  of  resources  to  meet  joint  capability  needs  and  balance  resources  and  risks.  The 

Joint  Staff  is  taking  steps  to  improve  prioritization  by  analyzing  joint  requirements  in  the 
context  of  alternative  existing  and  proposed  programs  within  capability  areas. 

Key  GAO  observations 

As  required  by  law  and  reflected  in  Joint  Staff  guidance,  the  JROC  has  the  responsibility 
to  assist  the  Chairman  of  the  Joint  Chiefs  of  Staff  in  developing  and  assigning  priority 
levels  for  joint  military  requirements  and  to  help  ensure  that  resource  levels  associated 
with  those  requirements  are  consistent  with  the  level  of  priority.  However,  the  JROC  has 
not  explicitly  considered  requirements’  affordability  in  the  context  of  DOD’s  budget.  As  we 
have  reported  since  2008,  the  JROC  has  not  prioritized  requirements,  considered 
redundancies  across  proposed  programs,  or  prioritized  and  analyzed  capability  gaps 
consistently  through  JCIDS.  A  key  reason  it  did  not  prioritize  needs  was  that  the  JROC 
reviewed  requirements  separately  and  largely  left  prioritization  and  trade-off  decisions  to 
the  military  services.3  Further,  DOD’s  ability  to  use  JCIDS  to  ensure  the  efficient  use  of 
resources  within  and  among  the  services  has  been  hampered  by  the  JROC’s  inability  to 
enforce  joint  priorities  in  service  budget  decisions.  According  to  officials,  the  Joint  Staff 
has  recently  initiated  efforts  to  assess  the  importance  of  each  capability  proposal  JCIDS 
reviews  in  relation  to  existing  capabilities  and  other  related  capability  needs  through  a 
“capability  portfolio”  approach.  Criteria  for  validation  of  a  proposal  would  include  the 
uniqueness  of  the  proposal  and  its  costs  and  benefits.  JCIDS’s  Functional  Capabilities 
Boards  are  already  providing  this  assessment  to  the  JROC  for  its  consideration  in  closed 
meetings.  However,  guidance  on  how  the  Functional  Capabilities  Boards  are  to  assess 
priorities  among  capabilities  is  not  specific  as  to  how  to  identify  joint  priorities.  See  the 
section  of  our  letter  on  the  Joint  Staffs  efforts  to  prioritize  capability  needs  for  additional 
information. 

GAO  scope  and  methodology 

To  evaluate  the  effectiveness  of  the  JCIDS  process  in  efficiently  investing  DOD’s 
resources,  we  reviewed  relevant  legislation  and  policy  documents  and  guidance  and 
discussed  issues  with  senior  DOD  officials.  We  also  reviewed  previous  GAO  assessments 
of  JCIDS  and  acquisition  issues  detailed  in  multiple  reports,  including  GAO-11-502; 

Defense  Acquisitions:  DOD’s  Requirements  Determination  Process  Has  Not  Been 

Effective  in  Prioritizing  Joint  Capabilities,  GAO-08-1060  (Washington,  D.C.:  Sept.  25, 

2008);  and  High-Risk  Series:  An  Update,  GAO-11-278  (Washington,  D.C.:  February 

2011). 

Sources:  DOD  and  GAO. 

aGAO,  DOD  Weapon  Systems:  Missed  Trade-off  Opportunities  During  Requirements  Reviews, 
GAO-1 1-502  (Washington,  D.C.:  June  16,  2011). 
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Table  5:  Section  862(b)(1)(C) — Evaluate  the  Effectiveness  of  JCIDS  in  Achieving  Control  of  Requirements  Creep — and  Section 
862(b)(1)(D) — Evaluate  the  Effectiveness  of  JCIDS  in  Achieving  Responsiveness  to  Changes  Occurring  after  the  Approval  of  a 
Requirements  Document  (Including  Changes  to  the  Threat  Environment,  the  Emergence  of  New  Capabilities,  or  Changes  in 
the  Resources  Estimated  to  Procure  or  Sustain  a  Capability) 


Summary  assessment 

The  JCIDS  process  does  not  actively  prevent  or  encourage  requirements  creep,3  as  these 
changes  are  driven  by  the  program  sponsor  and  may  not  appear  during  review  of 
documents. 

Key  GAO  observations 

We  have  previously  reported  that  despite  the  importance  of  well-developed  and  stable 
requirements  for  obtaining  good  weapon  program  outcomes,  program  requirements  often 
change  during  development.  For  example,  the  threat  the  program  originally  addressed 
may  change,  or  the  user  and  acquisition  communities  may  change  opinions  on  what  is 
needed  from  a  program.13  Programs  that  modified  critical  system  characteristics  after 
development  start  experienced  higher  levels  of  cost  growth  and  longer  delays  in 
delivering  capabilities.  The  JROC  typically  oversees  changes  only  for  those  system 
characteristics  considered  critical  to  delivering  a  capability.0  In  March  2011,  we  reported 
that  21  of  39  major  defense  acquisition  programs  we  assessed  had  at  least  one  change  to 
a  critical  characteristic,  either  increasing  or  decreasing  a  requirement.11  Specifically,  10  of 
the  21  programs  reported  adding  or  enhancing  a  critical  characteristic;  3  reported 
reducing,  deferring,  or  deleting  a  critical  characteristic;  and  8  reported  making  both  types 
of  changes. 

The  acquisition  community  has  taken  some  steps  to  address  requirements  creep  and 
changes  made  to  a  program’s  attributes  by  establishing  configuration  steering  boards  for 
major  acquisition  programs  to  review  and  limit  requirements  changes  and  avoid  cost 
increases.  However,  as  we  reported  in  July  201 1 ,  individual  programs  varied  in  how  they 
used  these  boards  to  control  requirements  and  mitigate  cost  and  schedule  risks.® 

GAO  scope  and  methodology 

To  evaluate  the  effectiveness  of  JCIDS  in  controlling  requirements  creep  and  responding 
to  changes,  we  reviewed  policy  documents  on  the  joint  requirements  process  and  the 
acquisition  system  and  our  prior  reviews  of  JCIDS  and  acquisition  issues,  including 

GAO-1 1-233SP. 

Sources:  DOD  and  GAO. 

aWe  asked  officials  about  their  perspectives  on  the  responsiveness  of  JCIDS  to  changes  occurring 
after  the  approval  of  a  requirements  document  and  they  had  no  specific  insights  to  offer.  Other  than 
the  examples  presented  in  this  table,  neither  we  nor  officials  were  able  to  identify  specific  examples 
regarding  the  effectiveness  of  JCIDS  in  achieving  responsiveness  to  changes. 

bGAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-08-467SP 
(Washington,  D.C.:  Mar.  31, 2008). 

“Specifically,  changes  may  only  appear  before  the  JROC  and  the  Joint  Capabilities  Board  for 
validation  if  they  affect  the  key  performance  parameters,  key  system  attributes,  or  other  performance 
attributes.  Changes  to  characteristics  of  a  lower  priority  are  typically  addressed  by  the  military 
services.  JROC-interest  programs  with  approved  capability  development  and  production  documents 
must  return  to  the  JROC  if  they  experience  certain  cost  growth. 

dGAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-1 1-233SP 
(Washington,  D.C.:  Mar.  29,  2011). 

eGAO,  Defense  Acquisitions:  DOD  Can  Improve  Its  Management  of  Configuration  Steering  Boards, 
GAO-11-640  (Washington,  D.C.:  July  7,  2011). 
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Table  6:  Section  862(b)(1)(E) — Evaluate  the  Effectiveness  of  JCIDS  in  Achieving  Development  of  the  Personnel  Skills, 

Capacity,  and  Training  Needed  for  an  Effective  and  Efficient  Requirements  Process 

Summary  assessment 

In  response  to  a  requirement  in  the  John  Warner  National  Defense  Authorization  Act  for 
Fiscal  Year  2007, a  DOD  developed  and  implemented  a  training  program  for  personnel 
who  develop  requirements  for  major  defense  acquisition  programs.  DOD  is  currently 
planning  to  expand  the  training  program  to  personnel  involved  in  developing  joint 
requirements  for  programs  that  do  not  meet  the  threshold  to  be  major  defense  acquisition 
programs.  This  program  helps  to  develop  the  personnel  skills,  capacity,  and  training  for 
the  requirements  process.5 

Key  GAO  observations 

The  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics, 
working  with  the  Defense  Acquisition  University,  developed  the  requirements 
management  certification  training  program  to  enhance  the  ability  of  individuals  to  write, 
review,  and  validate  requirements.  Defense  Acquisition  University  officials  reported  that 
about  1 ,000  personnel  with  responsibility  for  generating  requirements  for  major  defense 
acquisition  programs  took  the  requirements  development  training  in  2008.  Since  then, 
training  has  been  available  to  individuals  involved  in  smaller  programs,  but  DOD  officials 
believe  that  the  acquisition  community’s  ability  to  identify  and  document  joint  requirements 
would  be  improved  if  all  staff  completed  the  training  and  applied  the  concepts  in 
developing  proposals. 

The  curriculum  addresses  multiple  topics  related  to  generating  requirements,  including  the 
JCIDS  process  for  reviewing  and  approving  documents,  the  collaboration  between  JCIDS 
and  the  Defense  Acquisition  System,  and  DOD’s  resource  allocation  process.  The 
program  presents  a  series  of  courses  that  can  be  tailored  to  each  individual’s  level  of 
involvement  in  the  requirements  generation  process.  The  Defense  Acquisition  University 
provides  the  training  courses  and  tracks  individual  completion  of  each  course,  according 
to  officials,  but  beyond  the  required  completion  of  DOD  training,  each  component 
determines  the  specific  steps  necessary  to  certify  requirements  managers,  such  as 
additional  training  and  on-the-job  experience. 

GAO  scope  and  methodology 

To  evaluate  the  effectiveness  of  JCIDS  in  developing  the  personnel  skills,  capacity,  and 
training  needed  for  an  effective  and  efficient  requirements  process,  we  reviewed  relevant 
legislation  and  Joint  Staff  policy  documents  to  identify  the  requirement  to  implement 
training  for  personnel  involved  with  developing  joint  requirements.  We  interviewed  officials 
from  the  Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 

Logistics;  the  Joint  Staff;  and  the  Defense  Acquisition  University  to  discuss  DOD’s 
approach  for  implementing  the  training,  and  we  interviewed  service  and  combatant 
command  officials  to  discuss  the  extent  to  which  the  required  training  had  been 
implemented. 

Sources:  DOD  and  GAO. 


aSee  Pub.  L.  No.  109-364,  §  801  (2006). 

bNeither  we  nor  officials  identified  any  specific  programs  under  JCIDS  itself  to  develop  the  personnel 
skills,  capacity,  and  training  for  the  requirements  process.  Officials  identified  no  specific  DOD-wide 
development  programs  other  than  the  one  described  in  the  table. 
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Table  7:  Section  862(b)(2)(A) — the  Time  That  Requirements  Documents  Take  to  Receive  Approval  through  JCIDS 


Summary  assessment 

JCIDS  review  of  documents  validated  from  May  1 ,  201 0  through  May  31 , 201 1  took  an 
average  of  9  months  to  receive  approval.  Joint  Staff  and  service  officials  believed  that  the 
length  of  the  review  period  was  generally  appropriate  given  the  cost  and  complexity  of  the 
capabilities  proposed.  New  procedures  designed  to  improve  responsiveness  set  goals  for 
the  duration  of  JCIDS  reviews,  and  are  based  on  the  urgency  of  the  need. 

Key  GAO  observations 

The  average  of  9  months  for  review  of  documents  within  JCIDS  is  consistent  with  our 
previous  findings.  We  reported  in  2008  that  it  took  9.5  months  on  average  for  the  JROC  to 
validate  capability  proposals  submitted  from  2003  through  2005  and  8  months  for 
submissions  from  2006  through  2008. a 

JCIDS  guidance  in  effect  through  December  201 1  specified  timeliness  goals  for  the  initial 
commenting  and  adjudication  phases  of  the  process,  but  not  for  the  later  review  phases. b 
However,  goals  for  the  commenting  and  adjudication  phases  were  not  always  met.  For 
most  programs  we  analyzed,  reviews  exceeded  timeliness  goals  for  the  commenting  or 
adjudication  periods,  or  both,  by  about  25  percent  or  more.  The  duration  of  the  later 
review  phases  varied  and  accounted  for  the  most  time  between  submission  of  a  proposal 
and  validation.  According  to  Joint  Staff  and  service  officials,  the  reasons  for  extended 
review  times  included  need  for  additional  information  from  the  documents’  service 
sponsors  and  the  lack  of  sponsor  personnel  available  to  address  comments  and  approve 
changes.  We  previously  reported  that  given  the  size,  complexity,  and  level  of  funding 
associated  with  a  major  weapon  system,  the  investment  of  time  for  a  thorough,  joint 
review  may  be  warranted.0 

GAO  scope  and  methodology 

To  evaluate  the  time  it  took  to  review  and  approve  requirements  documents  through 

JCIDS,  we  reviewed  policy  documents  on  the  JCIDS  process.  We  selected  a 
nonprobability  sample  of  12  requirements  documents — 4  initial  capabilities  documents,  5 
capability  development  documents,  and  3  capability  production  documents — for  major 
weapon  systemsd  validated  from  May  1, 2010,  through  May  31, 2011,  to  determine  the 
time  it  took  the  JROC  to  approve  them.  We  focused  on  major  weapon  systems  because 
they  were  reviewed  through  each  of  the  JCIDS  phases.  We  interviewed  Joint  Staff  and 
service  officials  to  discuss  the  time  frames,  including  factors  that  contributed  to  timelines. 

Sources:  DOD  and  GAO. 

aGAO,  Defense  Acquisitions:  DOD’s  Requirements  Determination  Process  Has  Not  Been  Effective  in 
Prioritizing  Joint  Capabilities,  GAO-08-1060  (Washington,  D.C.:  Sept.  25,  2008). 

bSee  JCIDS  Manual,  enc.  D  (Jan.  31,  2011).  These  reviews  are  performed  by  the  Functional  and 
Joint  Capabilities  Boards  and  the  JROC. 

CGAO-08-1060. 

dWe  selected  requirements  for  major  weapon  systems  because  these  requirements  are  validated  by 
the  JROC  whereas  requirements  for  small  programs  are  generally  delegated  to  lower  validation 
authorities. 
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Table  8:  Section  862(b)(2)(B) — the  Quality  of  Cost  Information  Considered  in  JCIDS  and  the  Extent  of  Its  Consideration 


Summary  assessment 

The  cost  information  considered  during  JCIDS  reviews  has  not  always  been  reliable,  and 
the  extent  to  which  JCIDS  has  considered  cost  information  for  proposed  programs  was 
limited  to  funding  considerations. 

Key  GAO  observations 

As  we  reported  in  June  201 1 ,  the  services  did  not  consistently  provide  high-quality 
resource  estimates  to  the  JROC  to  aid  its  validation  decisions  and  cost  estimates  were 
often  unreliable  when  assessed  against  best  practices  criteria.3  In  some  cases,  the  quality 
of  service  estimates  varied  from  those  validated  by  the  services’  cost  analysis  agencies  to 
those  that  represented  less  rigorous  rough-orders-of-magnitude.b  In  most  cases,  the 
services  had  not  effectively  conducted  uncertainty  and  sensitivity  analyses,  which 
establish  confidence  levels  for  resource  estimates,  based  on  the  knowledge  available,  or 
examined  the  effects  of  changing  assumptions  and  ground  rules,  all  of  which  could  further 
the  JROC’s  efforts  to  provide  a  sound  basis  for  making  cost,  schedule,  and  performance 
trade-off  decisions.  Moreover,  we  reported  that  the  JROC  had  not  explicitly  considered  a 
requirement’s  affordability  in  a  broader  context  during  JCIDS  reviews. 

Owing  to  the  poor  quality  of  the  estimates  of  programs  we  reviewed,  we  recommended  in 
the  June  201 1  report  that  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff  require  that 
capability  sponsors  present  resource  estimates  that  have  been  reviewed  by  a  military 
service’s  cost  analysis  organization  to  ensure  that  best  practices  are  being  followed.  The 
Joint  Staff  partially  concurred  with  our  recommendation,  stating  that  program  office  cost 
estimates  were  compared  to  independent  cost  estimates  during  reviews  of  capability 
development  documents.  However,  it  is  not  clear  whether  the  JCIDS  guidance  issued  in 
January  2012  fully  responds  to  our  recommendation  from  the  June  201 1  report.  Officials 
told  us  that  the  guidance  does  not  explicitly  address  the  reliability  of  cost  information,  but 
according  to  Joint  Staff  officials,  the  JROC  intends  to  rely  more  heavily  on  estimates 
developed  by  the  Director  of  Cost  Assessment  and  Program  Evaluation  to  inform  its 
deliberations. 

GAO  scope  and  methodology 

To  evaluate  the  quality  of  cost  information  considered  in  JCIDS  and  the  extent  of  its 
consideration,  we  relied  on  prior  GAO  work — GAO-1 1-502 — that  addressed  the  extent  to 
which  the  JROC  considered  cost  information.  We  also  interviewed  Joint  Staff  and  Office 
of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  officials,  and 
reviewed  JCIDS  guidance  to  characterize  the  JROC’s  approach  for  addressing  our  prior 
recommendation. 

Sources:  DOD  and  GAO. 

aGAO,  DOD  Weapon  Systems:  Missed  Trade-off  Opportunities  During  Requirements  Reviews, 
GAO-1 1-502  (Washington,  D.C.:  June  16,  2011). 

bA  rough-order-of-magnitude  estimate  is  a  quick,  high-level  estimate  that  generally  involves  less  time 
and  effort  than  a  budget-quality  estimate. 
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Table  9:  Section  862(b)(2)(C) — the  Extent  to  Which  JCIDS  Establishes  a  Meaningful  Level  of  Priority  for  Requirements 


Summary  assessment 

JCIDS  has  not  established  a  meaningful  level  of  priority  for  joint  requirements,  but  it  is 
adopting  a  new  approach  to  prioritize  joint  requirements  within  the  context  of  current 
capabilities. 

Key  GAO  observations 

In  2008,  we  reported  that  JCIDS  was  ineffective  in  setting  priorities  across  the  services 
and  that  virtually  all  capability  proposals  submitted  to  JCIDS  had  been  approved  or 
validated.3  In  June  201 1 ,  we  reported  that  the  JROC  still  did  not  prioritize  requirements 
during  the  JCIDS  process,  consider  redundancies  across  proposed  programs,  or  prioritize 
and  analyze  capability  gaps  in  a  consistent  manner b  DOD  officials  told  us  that 
prioritization  across  programs  occurred  primarily  through  DOD’s  budgeting  process,  which 
is  the  responsibility  of  the  military  services  and  the  Office  of  the  Under  Secretary  of 

Defense  (Comptroller).  JCIDS  guidance  in  effect  through  December  201 1  required  that 
capability  sponsors  prioritize  capability  gaps  and  identify  redundancies  when  developing 
initial  capabilities  documents,  but  it  did  not  specify  the  criteria  sponsors  should  use  to 
prioritize  needs.  Our  review  found  that  sponsors  used  different  methods,  with  some 
ranking  priorities  in  numerical  order  and  others  assigning  priority  categories.  Some  did  not 
report  on  redundancies  and  overlap. 

The  JROC  and  Joint  Staff  have  acknowledged  the  need  to  improve  the  JROC’s  ability  to 
prioritize  capabilities  through  the  JCIDS  process.  Officials  noted  that  the  Joint  Staff  is  now 
implementing  a  new  approach  to  impose  prioritization  of  joint  needs  within  the  context  of 
existing  capabilities  in  joint  capability  areas.  However,  the  new  approach  has  not  been 
fully  developed  and  clearly  documented,  and  the  extent  to  which  current  momentum 
toward  assigning  priorities  will  be  sustained  is  uncertain.  See  the  section  of  our  letter  on 
the  Joint  Staff’s  efforts  to  prioritize  capability  needs  for  more  information. 

GAO  scope  and  methodology 

To  evaluate  the  extent  to  which  JCIDS  establishes  a  meaningful  level  of  priority  for 
requirements,  we  reviewed  law  and  Joint  Staff  policy  documents  to  understand  the  roles 
and  requirements  of  the  JCIDS  process  and  the  JROC  as  it  pertains  to  prioritization.  We 
interviewed  Joint  Staff  officials  to  discuss  changes  to  the  approach  for  prioritizing 
capability  requirements.  We  also  reviewed  prior  GAO  reports,  including  GAO-1 1-502  and 
GAO-08-1060,  that  discussed  prioritization  and  compared  the  current  efforts  to  prioritize 
with  what  was  reported  in  the  past. 

Sources:  DOD  and  GAO. 

aGAO,  Defense  Acquisitions:  DOD’s  Requirements  Determination  Process  Has  Not  Been  Effective  in 
Prioritizing  Joint  Capabilities,  GAO-08-1060  (Washington,  D.C.:  Sept.  25,  2008).  GAO’s  2008  report 
assessed  JCIDS  under  guidance  prior  to  CJCSI  31 70.01  G. 

bGAO,  DOD  Weapon  Systems:  Missed  Trade-off  Opportunities  During  Requirements  Reviews, 
GAO-11-502  (Washington,  D.C.:  June  16,  2011). 
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Table  10:  Section  862(b)(2)(D) — the  Extent  to  Which  JCIDS  Is  Considering  Trade-offs  between  Cost,  Schedule,  and 
Performance  Objectives 


Summary  assessment 

The  JROC  has  not  always  considered  trade-offs  among  cost,  schedule,  and  performance 
objectives  during  JCIDS.  It  is  premature  to  assess  current  initiatives  for  considering  trade¬ 
offs  within  JCIDS. 

Key  GAO  observations 

The  Weapon  Systems  Acquisition  Reform  Act  of  2009  amended  section  181  to  require  the 
JROC  to  assist  the  Chairman  of  the  Joint  Chiefs  of  Staff  in  ensuring  the  consideration  of 
trade-offs  among  cost,  schedule,  and  performance  objectives  for  joint  military 
requirements.3  However,  as  we  reported  in  June  201 1 ,  most  trade-offs  are  made  by  the 
military  services  outside  the  JCIDS  process  as  they  analyze  alternative  solutions  for 
meeting  capability  needs.  The  JROC’s  reviews  of  requirements  do  not  align  with  these 
activities,  and  as  a  result,  the  JROC  does  not  have  an  opportunity  to  provide  the  services 
input  on  trade-offs  or  on  the  proposed  solution  to  a  capability  gap  before  it  is  selected, 
and  significant  time  and  resources  may  be  expended  before  the  JROC  gets  to  formally 
weigh  in  during  the  JCIDS  process.  However,  we  discussed  several  instances  where  the 
JROC  held  requirements  firm  during  reviews  of  programs  reporting  substantial  cost 
growth.  According  to  officials,  by  accepting  increased  cost  and  schedule  delays,  the 

JROC  essentially  traded  cost  and  possibly  schedule  for  performance.  We  made 
recommendations  to  improve  the  JROC’s  ability  to  make  trade-offs,  including  establishing 
a  mechanism  to  review  the  results  of  the  services’  alternatives  earlier  in  the  process, 
among  other  actions. b 

The  Joint  Staff  issued  new  JCIDS  guidance  intended  to  better  position  the  JROC  for 
considering  cost,  schedule,  and  performance  trade-offs  and  provide  improved  accuracy 
early  in  the  process.  As  explained  in  DOD  documentation,  changes  include  the  following: 

•  Conducting  a  review  of  the  program  sponsor’s  analysis  of  alternatives  study — 
including  study  results,  such  as  cost-,  schedule-,  and  performance-level 
recommendations — prior  to  an  acquisition  program  proceeding  to  the  next  phase  of 
development. 

•  Considering  cost,  schedule,  performance,  and  quantity  targets  in  making  validation 
decisions  and  annotating  decisions  in  JROC  memorandums,  as  appropriate. 

•  Reviewing  requirements  that  have  unacceptably  deviated  from  schedule,  quantity, 
cost,  and  performance  parameters. 

GAO  scope  and  methodology 

To  evaluate  the  extent  to  which  JCIDS  is  considering  trade-offs  between  cost,  schedule, 
and  performance  objectives,  we  relied  primarily  on  GAO’s  June  201 1  report — 

GAO-11-502 — that  discussed  the  extent  to  which  the  JROC  has  considered  trade-offs 
within  programs.  We  also  reviewed  Joint  Staff  briefings  and  guidance  documents  to 
identify  changes  in  guidance  with  regard  to  cost,  schedule,  and  performance  trade-offs 
since  June  2011. 

Sources:  DOD  and  GAO. 

aSee  Pub.  L.  No.  111-23,  §  201(b)(1)(C)  (2009)  (codified  at  10  U.S.C.  §  181(b)(1)(C)).  Because  of  this 
statutory  responsibility,  we  focused  on  the  JROC. 

bGAO,  DOD  Weapon  Systems:  Missed  Trade-off  Opportunities  During  Requirements  Reviews, 
GAO-11-502  (Washington,  D.C.:  June  16,  2011). 
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Table  11:  Section  862(b)(2)(E) — the  Quality  of  Information  on  Sustainment  Considered  in  JCIDS  and  the  Extent  to  Which 
Sustainment  Information  Is  Considered 


Summary  assessment 

The  quality  of  information  on  sustainment  planning  that  program  sponsors  provided 

JCIDS  varied  and  the  JROC  has  not  systematically  considered  sustainment  before 
validating  programs.3 

Key  GAO  observations 

The  JCIDS  Manual  in  effect  through  December  201 1  required  that  program  sponsors  of 
major  defense  acquisition  programs  develop  information  on  sustainment  based  on  four 
factors — materiel  availability,  operational  availability,  reliability,  and  ownership  cost3 — and 
listed  descriptive  criteria,  against  which  documents  would  be  reviewed,  for  program 
sponsors  to  consider.*1  We  have  previously  reported  that  the  rigor  of  the  analysis 
supporting  sustainment  planning  and  cost  estimates  varied.  Our  current  review  of  six 
capability  development  documents  showed  that  all  the  sponsors  addressed  the 
information  required,  but  the  level  of  detail  varied.  For  example,  the  Navy  provided  an 
ownership  cost  estimate  for  its  proposed  Littoral  Combat  Ship  capability  but  not 
information  to  support  the  estimate  or  a  description  of  plans  to  monitor,  collect,  or  validate 
the  ownership  cost  data.  Further,  the  level  of  detail  provided  for  the  supporting  criteria 
varied,  with  some  documents  providing  a  paragraph  of  supporting  information  for  some 
criteria  and  others  providing  single-sentence  responses.  Additionally,  none  of  the 
documents  we  reviewed  included  a  clear  definition  of  the  type  of  cost  structure  used  to 
determine  the  ownership  cost  estimates  for  the  proposed  programs. 

JCIDS  has  not  generally  sought  detailed  information  to  present  to  the  JROC  for 
consideration  before  validating  proposals.  Although  the  review  criteria  associated  with 
each  of  the  four  required  sustainment  metrics  may  not  be  applicable  to  every  program, 
according  to  DOD  officials  the  information  could  be  useful  for  JROC  validation  discussions 
and  decisions.  Without  complete  information,  the  JROC  may  not  always  have  the 
sustainment  information  it  needs  to  effectively  manage  and  validate  capability  documents. 
See  the  section  of  our  letter  on  sustainment  information  available  to  the  JROC  for  more 
information. 

GAO  scope  and  methodology 

To  assess  the  quality  of  reported  sustainment  information  and  the  extent  to  which  it  is 
being  considered  in  JCIDS,  we  worked  with  DOD  officials  to  identify  six  capability 
development  documents  that  included  sustainment  information.  We  reviewed  the 
documents  to  determine  whether  and  the  extent  to  which  sustainment  information  was 
included  and  interviewed  DOD  officials  to  discuss  reported  sustainment  data.  We  also 
reviewed  relevant  law  and  Joint  Staff  policy  documents  to  understand  reporting 
requirements. 

Sources:  DOD  and  GAO. 

aUnder  a  version  of  the  JCIDS  Manual  in  effect  through  December  201 1 ,  the  four  sustainment  factors 
were  required  to  be  developed  for  major  defense  acquisition  programs.  Applicability  of  some  of  the 
information  for  acquisition  category  II  and  lower  programs  was  left  to  the  determination  of  the 
sponsor.  Under  the  current  version  of  the  manual,  issued  January  19,  2012,  acquisition  category  II 
and  lower  programs,  with  materiel  solutions,  are  to  include  either  the  sustainment  metric  or  sponsor- 
defined  sustainment  metrics.  As  a  consequence,  under  both  versions,  sustainment  information  may 
not  necessarily  be  considered  or  have  been  considered  outside  the  context  of  major  defense 
acquisition  programs. 

bThe  JCIDS  Manual  required  the  inclusion  of  sustainment  information  in  capability  development 
documents  beginning  in  2007. 

cThe  current  version  of  the  JCIDS  Manual  renamed  the  fourth  factor  as  “operation  and  support  cost.” 
dSee  JCIDS  Manual,  enc.  B  (Jan.  31 ,  201 1 ). 
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Table  12:  Section  862(b)(2)(F) — an  Evaluation  of  the  Advantages  and  Disadvantages  of  Designating  a  Commander  of  a  Unified 
Combatant  Command  for  Each  JROC-Interest  Requirements  Document  to  Provide  a  Joint  Evaluation  Task  Force  for  Certain 
Purposes 


Summary  assessment 

Developing  a  joint  evaluation  task  force  could  have  some  advantages  for  combatant 
commands,  such  as  advancing  a  dialogue  among  the  commands,  but  the  disadvantages 
of  burdening  commands  with  additional  duties  of  participation  may  outweigh  the  benefits. 

Key  GAO  observations 

Combatant  command  officials  reported  that  participation  in  a  task  force  might  provide 
greater  opportunities  for  the  commands  to  provide  input,  but  they  noted  that  the 
commands  already  participate  in  identification  and  discussion  of  capability  requirements 
through  the  current  JCIDS  process  and  other  forums.3  Specifically,  commands  can 
participate  in 

•  experimentation  and  testing  efforts  and  analysis  of  alternative  teams; 

•  discussions  of  capability  proposals  that  deviate  from  cost,  schedule,  or  quantity 
targets  in  existing  forums; 

•  advocating  for  desired  material  solutions  as  end  users  through  integrated  priority  lists 
and  service  component  collaboration;  and 

•  commenting  on  concept  of  operations  and  doctrine  through  participation  in  regular 
Functional  Capabilities  Board  meetings. 

DOD  officials  consistently  cited  the  following  three  key  disadvantages  to  formation  of  joint 
evaluation  task  forces: 

•  Implementation  of  the  task  force  would  represent  a  shift  in  responsibilities  from  the 
military  services — which  have  historically  completed  these  duties — to  the  combatant 
commands. 

•  Combatant  commands  have  limited  manpower  and  few  acquisition-trained  specialists 
to  appropriately  fill  the  role  of  combat  developer. 

•  It  is  unclear  how  the  leadership  responsibilities  would  be  divided  among  the 
combatant  commands,  particularly  where  a  solution  is  of  interest  to  multiple 
commands. 

In  response  to  congressional  questions  about  establishing  such  a  task  force,  the  current 
Vice  Chairman  of  the  Joint  Chiefs  of  Staff  stated  that  such  a  task  force  could  result  in  an 
additional  layer  of  coordination  and  staffing  between  the  existing  requirements  and 
acquisition  processes,  further  delaying  the  identification  of  capability  gaps  and 
subsequent  fielding  of  systems  addressing  those  gaps. 

GAO  scope  and  methodology 

To  evaluate  the  advantages  and  disadvantages  of  implementing  a  joint  evaluation  task 
force,  we  reviewed  a  March  2010  report  prepared  by  the  House  Armed  Services 

Committee  Panel  on  Defense  Acquisition  Reform  and  interviewed  Joint  Staff  and 
combatant  command  officials  to  discuss  advantages  and  disadvantages  of  the  concept. 

Sources:  DOD  and  GAO. 

Note:  See  the  introduction  to  this  appendix  for  the  full  text  of  this  requirement,  including  the  specific 
purposes  identified. 

aAmong  the  findings  of  a  May  201 1  GAO  report  was  that  combatant  commands  noted  the  importance 
of  participating  in  the  development  of  joint  requirements,  but  questioned  the  value  of  what  they 
described  as  a  resource-intensive  and  time-consuming  process  that  is  not  always  responsive  to  their 
more  immediate  capability  needs.  They  also  questioned  the  value  of  a  process  that  results  in 
decisions  that  while  influential,  are  advisory  to  the  acquisition  and  budget  process  driven  by  service 
investment  priorities.  See  GAO,  Defense  Management:  Perspectives  on  the  Involvement  of 
Combatant  Commands  in  the  Development  of  Joint  Requirements,  GAO-1 1-527R  (Washington,  D.C.: 
May  20,  201 1 ).  Although  the  report  did  not  address  involvement  in  a  joint  evaluation  task  force,  it 
provides  combatant  command  perspectives  on  the  overall  JCIDS  process,  including  views  on  greater 
combatant  command  involvement. 
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To  assess  the  extent  to  which  the  Joint  Staff  has  developed  and 
implemented  an  analytic  approach  to  prioritize  capability  needs,  we 
reviewed  relevant  legislation  and  Joint  Staff  guidance  on  the  roles  and 
requirements  of  the  JCIDS  process  and  the  JROC  as  it  pertains  to 
assigning  levels  of  priority  to  capability  proposals.  Specifically,  we 
reviewed  section  181  of  the  U.S.  Code,  Title  10;  Chairman  of  the  Joint 
Chiefs  of  Staff  Instruction  31 70.01  G;  Chairman  of  the  Joint  Chiefs  of  Staff 
Instruction  31 70.01  H  (which  we  reviewed  in  draft  form);  and  the  JCIDS 
Manual.  We  then  compared  the  current  and  prior  versions  of  the 
instruction  and  manual  to  identify  changes  in  the  guidance  with  respect  to 
prioritization  of  capability  proposals.  We  met  with  officials  from  the  Joint 
Staff;  Department  of  the  Air  Force;  Department  of  the  Army;  Department 
of  the  Navy;  and  Office  of  the  Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  to  discuss  their  perspectives  on  the 
implementation  of  changes  to  JCIDS  with  respect  to  prioritizing  capability 
requirements.  In  order  to  understand  how  the  JROC  is  implementing  its 
new  approach  for  prioritizing  capabilities,  we  reviewed  briefing  materials 
presented  at  a  JROC  forum  in  November  201 1 .  To  corroborate  our 
understanding  of  the  documents  we  reviewed,  we  conducted  interviews 
with  Joint  Staff  and  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology  and  Logistics  officials.  To  understand  the  Joint 
Staff’s  recent  internal  review  of  JCIDS,  we  reviewed  the  charter  and 
recommendations,  and  met  with  Joint  Staff  officials  to  discuss  how  those 
recommendations  from  the  review  might  affect  the  JROC’s  prioritization  of 
capability  proposals.  We  also  reviewed  prior  reports  by  GAO,  the  House 
Armed  Services  Committee,  and  the  Defense  Business  Board  that 
discussed  prioritization  of  capability  proposals  through  JCIDS,  and 
compared  the  JROC’s  current  efforts  to  prioritize  with  what  has  been 
reported  in  the  past.  We  assessed  whether  the  guidance  in  the  JCIDS 
Manual  and  JCIDS  instruction  (in  draft  form  during  our  review  and  issued 
in  January  2012)  on  prioritization  meets  the  intent  of  recommendations 
contained  in  our  prior  reports. 

To  assess  the  extent  to  which  the  JROC  has  considered  aspects  of  the 
availability  and  operational  support  requirements  of  weapon  systems — 
called  sustainment — when  validating  the  requirements  of  proposed 
capability  solutions,  we  reviewed  relevant  DOD  and  Joint  Staff  policy 
documents  and  related  guidance  outlining  the  requirement  to  develop  and 
report  sustainment  metrics  for  capability  documents.  Specifically,  we 
reviewed  the  reporting  requirements  for  major  defense  acquisition 
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programs  processed  through  JCIDS.1  To  determine  JCIDS  reporting 
requirements  for  capability  development  documents  and  to  understand 
the  JCIDS  process,  we  reviewed  the  JCIDS  Manual  enclosure  pertaining 
to  sustainment  and  instructions  from  the  Chairman  of  the  Joint  Chiefs  of 
Staff.  We  also  reviewed  prior  GAO  work  on  related  topics.  Further,  we 
interviewed  DOD  and  Joint  Staff  officials  to  discuss  preparation, 
presentation,  and  consideration  of  sustainment  data. 

We  also  conducted  a  case  study  analysis  of  select  capability 
development  documents  that  included  sustainment  information.  We 
sought  a  universe  of  all  capability  development  documents  subject  to 
reporting  the  sustainment  key  performance  parameter  validated  since 
2007,  when  JCIDS  began  requiring  program  sponsors  to  include  this 
information.  We  initially  obtained  a  universe  of  22  JCIDS  capability 
development  documents  from  the  Office  of  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology  and  Logistics/Logistics  and  Materiel 
Readiness.  We  relied  on  this  provided  list  of  documents  because  the 
Joint  Staff’s  Knowledge  Management/Decision  Support  database  did  not 
produce  reliable  results  of  all  requirements  documents  containing 
sustainment  information.  We  narrowed  the  list  by  eliminating  programs 
that  have  been  downgraded,  truncated,  or  canceled;  programs  in  which 
the  sustainment  data  were  not  at  the  key  performance  parameter  level  or 
were  more  narrowly  defined  by  only  one  type  of  platform;  programs 
whose  information  was  contained  in  capability  production  documents 
rather  than  capability  development  documents;  and  programs  whose 
capability  development  documents  were  entered  in  JCIDS  before  the  full 
implementation  of  the  sustainment  key  performance  parameter 
requirement.  Additionally,  several  programs  did  not  have  supporting 
documentation  that  would  allow  a  review;  Joint  Staff  officials  we  met  with 
stated  that  these  documents  were  not  available  either  because  of  a 
misidentification  of  the  type  of  capability  document  that  was  being 
reviewed  (a  capability  production  document  as  opposed  to  a  capability 
development  document),  or  because  the  document  was  not  included  in 
the  Joint  Staff’s  database  for  JROC  review.  These  factors  led  to  a  refined 


1  Under  the  version  of  the  JCIDS  Manual  in  effect  during  the  time  of  our  review, 
sustainment  metrics  were  to  be  developed  for  all  acquisition  category  I  programs  involving 
materiel  solutions  and  acquisition  category  II  and  below  programs  as  determined  by  the 
sponsor.  See  JCIDS  Manual,  app.  B  to  enc.  B,  para.  2  (Jan.  31 , 201 1 ).  The  current 
version  of  the  manual  states  that  acquisition  category  II  and  below  programs,  with  materiel 
solutions,  shall  include  the  sustainment  metrics  or  sponsor-defined  sustainment  metrics. 
See  JCIDS  Manual,  app.  E  to  enc.  B,  para.  2  (Jan.  19.  2012). 
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universe  of  12  requirements  documents.  We  then  randomly  selected 
capability  development  documents  for  two  programs  per  service — Army, 
Air  Force,  and  Navy — resulting  in  a  total  of  six  programs  to  serve  as  case 
studies.  Because  this  was  a  nonprobability  sample  of  programs,  the 
results  are  not  generalizable  to  all  programs;  however,  they  are  illustrative 
of  the  kinds  of  issues  that  are  possible  in  such  programs.  In  order  to 
assess  reported  sustainment  information  in  the  six  selected  cases,  we 
performed  a  content  analysis  of  the  documentation  available  for  the  six 
cases.  Two  GAO  analysts  independently  reviewed  each  of  the  six 
capability  development  documents,  assessing  whether  each  of  the 
individual  elements  of  the  JCIDS  Manual  sustainment  metrics  was 
included,  coding  the  inclusion  of  each  metric  as  “yes,”  “no,”  “partial,”  and 
“don’t  know.”  The  two  analysts  then  discussed  and  reconciled  all  initial 
disagreements  regarding  the  assigned  codes.  We  then  discussed  the 
results  of  this  content  analysis  with  officials  from  the  Joint  Staff  and  the 
Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology  and 
Logistics/Logistics  and  Materiel  Readiness  to  verify  that  the  results  of  our 
analysis  were  valid. 

We  conducted  this  performance  audit  from  April  201 1  through  February 
2012  in  accordance  with  generally  accepted  government  auditing 
standards.  Those  standards  require  that  we  plan  and  perform  the  audit  to 
obtain  sufficient,  appropriate  evidence  to  provide  a  reasonable  basis  for 
our  findings  and  conclusions  based  on  our  audit  objectives.  We  believe 
that  the  evidence  obtained  provides  a  reasonable  basis  for  our  findings 
and  conclusions  based  on  our  audit  objectives. 
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the  joint  staff 

WASHINGTON,  BC 

8  February  2012 


Reply  ZIP  Code: 

20318-0300 

Mr.  John  H.  Pendleton 
Director,  Defense  Capabilities  and 
Management 

U.S.  Government  Accountability 
Office 

441  G  Street,  NW 
Washington,  D.C.  20548 

Dear  Mr.  Pendleton: 

Thank  you  for  the  opportunity  to  review  GAO  Draft  Report  GAO- 12-339, 
‘Defense  Management:  Guidance  and  Progress  Measures  are  Needed  to  Realize 
Benefits  From  Changes  in  DOD’s  Joint  Requirements  Process,’  dated 
6  January  2012  (GAO  Code  351609).  The  DOD  response  to  the  report’s 
recommendations  is  enclosed. 

I  believe  the  report  reflects  a  fair  and  objective  assessment  of  the  current 
Joint  Staffs  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
process  during  the  GAO  assessment  period.  It  appropriately  highlights  the 
active  efforts  to  improve  the  JCIDS/  Joint  Requirements  Oversight  Council 
process  and  procedures  to  include  recently  updated  guidance  in  Chairman  of 
the  Joint  Chiefs  of  Stafflnstruction  (CJCSI)  3170.01H,  CJCSI  5123.01F,  and 
the  JCIDS  Manual. 

The  Joint  Staff  point  of  contact  is  Mr.  Randolph  Wood;  J-8/Requirements 
Management  Division;  703-614-9628;  randolph.wood@js.pentagon.mil. 


CRAIG  A.  FRANKLIN 
Major  General,  USAF 
Vice  Director,  Joint  Staff 


Enclosure 
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ENCLOSURE 


GAO  DRAFT  REPORT  DATED  6  JANUARY  2012 
GAO-12-339  (GAO  CODE  351609) 

“DEFENSE  MANAGEMENT:  GUIDANCE  AND  PROGRESS  MEASURES  ARE 
NEEDED  TO  REALIZE  BENEFITS  FROM  CHANGES  IN  DOD’S  JOINT 
REQUIREMENTS  PROCESS” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
TO  THE  GAO  RECOMMENDATIONS 

RECOMMENDATION  1:  The  Government  Accountability  Office  (GAO) 
recommends  that  the  Vice  Chairman  of  the  Joint  Chiefs  of  Staff,  as  the  Joint 
Requirements  Oversight  Council  (JROC)  Chairman,  revise  and  implement 
guidance  to  reflect  changes  to  the  Joint  Capabilities  Integration  and 
Development  System  (JCIDS)  process  as  well  as  to  establish  criteria  and 
measures  for  determining  the  relative  importance  of  capability  needs  across 
capability  areas  and  assessing  progress. 

DOD  RESPONSE:  Partially  concur. 

Substantial  changes  to  the  joint  requirements  development  process  have 
recently  been  made  after  more  than  a  year  of  review,  discussions,  and 
deliberations.  With  regard  to  prioritization  of  capability  needs  and  assessing 
progress,  the  Director,  Joint  Staff  recently  approved  and  signed  the  JCIDS 
instruction  and  JROC  Charter  instruction  (10  January  2012).  Changes 
outlined  in  these  instructions  and  further  detailed  in  the  revised  JCIDS  Manual 
(19  January  2012)  provide  first  steps  towards  an  improved  prioritization 
methodology  by  the  Functional  Capability  Boards,  Joint  Capabilities  Board, 
and  JROC  to  take  into  account  capability  requirements  in  an  expanded 
portfolio  view.  Capability  gap  and  solution  discussions  presented  for  the  JROC 
will  incorporate  an  evolving  portfolio  assessment  tool  that  provides  a  view  of 
the  proposed  gap/ solution  with  regard  to  its  strategic  guidance  importance 
along  With  other  similar  solutions  already  being  developed  or  available  to  the 
joint  force. 

In  addition  to  the  DOD  priorities  presented  in  the  recently  released  Strategic 
Guidance  for  the  Department  (Januaty  2012),  the  revised  Capability  Gap 
Assessment/Integrated  Priority  List  (CGA/IPL)  process  will  provide  further 
direction  for  sponsors  and  other  JCIDS  implementing  bodies  to  focus  on  the 
most  important  areas  as  capability  gaps/ solutions  are  proposed,  reviewed,  and 
validated,  or  not.  During  this  current  review  of  Combatant  Command  (CCMD) 
CGA/IPL  inputs  (for  FYs  14-18),  the  process  includes  the  incorporation  of  the 
Chairman’s  Risk  Assessment  (CRA).  By  associating  the  capability  gaps 
identified  in  the  CCMD  IPLs  with  strategic  and  militaiy  risks  from  the  CRA,  the 
JROC  will  be  able  to  prioritize  and  make  informed  decisions  as  to  which  gaps 
are  the  most  critical  and  which  are  not  of  sufficient  priority  to  justify  additional 
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actions  or  resources.  This  evaluation  and  the  decisions  resulting  from  this 
revised  CGA/IPL  process  should  further  inform  and  assist  in  future  JCIDS 
prioritization  methodology  and  results. 

RECOMMENDATIOH  2:  The  GAO  recommends  that  the  Vice  Chairman  of  the 
Joint  Chiefs  of  Staff,  as  the  JROC  Chairman,  explicitly  require  that  program 
sponsors  address  each  of  the  criteria  outlined  for  the  individual  sustainment 
metrics  when  submitting  capability  development  documents. 

DOD  RESPONSE:  Partially  concur. 

The  criteria  outlined  for  the  sustainment  metric  key  performance  parameter 
and  key  system  attributes  within  JCIDS  were  designed  to  be  a  “shaping” 
mechanism  for  the  requirements  community.  They  were  not  designed  to  be 
prescriptive.  Each  program  is  unique,  and  criteria  applicable  to  one  program 
may  not  be  applicable  to  another.  The  individual  criteria  are  a  guideline  to 
ensure  critical  thinking/evaluation  has  occurred  in  development  and  review  of 
a  program’s  sustainment  metric.  Capability  development  documents  (CDDs) 
are  vetted  through  the  Joint  Staff  and  Office  of  the  Secretary  of  Defense  (OSD), 
providing  opportunities  to  address  omissions/discrepancies.  Reviews  are  held 
internal  to  OSD  as  well  as  with  program  sponsors.  These  reviews  enable  the 
Joint  Staff  to  ensure  a  complete  and  thoroughly  vetted  document  is  presented 
to  the  JROC  for  requirements  approval/ validation  of  the  proposed  capability 
solution.  Additionally,  the  analyses  and  rationale  for  metric  development 
resides  in  documents  other  than  the  CDD.  Documents  such  as  the  Reliability, 
Availability,  Maintainability,  and  Cost  report.  Life  Cycle  Support  Plan, 
Independent  Cost  Estimate,  and  Program  Life  Cycle  Cost  Estimate  contain 
more  detailed  documentation  supporting  the  development  of  the  sustainment 
metric.  DOD  policy  discourages  replication  of  data  and  analysis  across 
multiple  documents. 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and 
policies;  and  provides  analyses,  recommendations,  and  other  assistance 
to  help  Congress  make  informed  oversight,  policy,  and  funding  decisions. 
GAO’s  commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no 
cost  is  through  GAO’s  website  (www.gao.gov).  Each  weekday  afternoon, 
GAO  posts  on  its  website  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  website, 

http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

Connect  with  GAO 

Connect  with  GAO  on  Facebook,  Flickr,  Twitter,  and  YouTube. 

Subscribe  to  our  RSS  Feeds  or  E-mail  Updates.  Listen  to  our  Podcasts. 
Visit  GAO  on  the  web  at  www.gao.gov. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Website:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Katherine  Siggerud,  Managing  Director,  siggerudk@gao.gov,  (202)  512- 
4400,  U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room 
7125,  Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngd@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 

W 

Please  Print  on  Recycled  Paper. 

